I am seeing extremely slow performance with glusterfs. OS: CentOS 5 glusterfs version: glusterfs-1.3.9-1 Server configuration: ############################################## ### GlusterFS Server Volume Specification ## ############################################## #### CONFIG FILE RULES: ### "#" is comment character. ### - Config file is case sensitive ### - Options within a volume block can be in any order. ### - Spaces or tabs are used as delimitter within a line. ### - Multiple values to options will be : delimitted. ### - Each option should end within a line. ### - Missing or commented fields will assume default values. ### - Blank/commented lines are allowed. ### - Sub-volumes should already be defined above before referring. volume shelf-1 type storage/posix option directory /glusterfs/shelf-1 end-volume volume shelf-1-readahead type performance/read-ahead subvolumes shelf-1 end-volume volume shelf-1-writebehind type performance/write-behind option aggregate-size 1MB option window-size 4MB subvolumes shelf-1-readahead end-volume ### Add network serving capability to above brick. volume server type protocol/server option transport-type tcp/server # For TCP/IP transport subvolumes shelf-1-writebehind option auth.ip.shelf-1-writebehind.allow 127.0.0.1 end-volume Local client configuration: ############################################## ### GlusterFS Client Volume Specification ## ############################################## volume storage type protocol/client option transport-type tcp/client option remote-host 127.0.0.1 option remote-subvolume shelf-1-writebehind end-volume Writing directly to the underlying filesystem: -bash-3.2# dd if=/dev/zero of=/glusterfs/shelf-1/testing count=20M 20971520+0 records in 20971520+0 records out 10737418240 bytes (11 GB) copied, 230.862 seconds, 46.5 MB/s Writing through glusterfs with no readahead/writebehind: -bash-3.2# dd if=/dev/zero of=/storage/testing count=20M 20971520+0 records in 20971520+0 records out 10737418240 bytes (11 GB) copied, 3429.89 seconds, 3.1 MB/s Using the above configuration is even slower: -bash-3.2# dd if=/dev/zero of=/storage/testing count=20M 20971520+0 records in 20971520+0 records out 10737418240 bytes (11 GB) copied, 4379.16 seconds, 2.5 MB/s What should I be trying to improve performance? I really need help! -- Matt
Matt, Please try dd with a larger block (bs=16k). The default block size of dd is 512 bytes which is very less for network filesystems. thanks, avati 2008/12/13 Matt Lawrence <matt.lawrence at tamu.edu>:> I am seeing extremely slow performance with glusterfs. > > OS: CentOS 5 > glusterfs version: glusterfs-1.3.9-1 > > Server configuration: > > ############################################## > ### GlusterFS Server Volume Specification ## > ############################################## > > #### CONFIG FILE RULES: > ### "#" is comment character. > ### - Config file is case sensitive > ### - Options within a volume block can be in any order. > ### - Spaces or tabs are used as delimitter within a line. > ### - Multiple values to options will be : delimitted. > ### - Each option should end within a line. > ### - Missing or commented fields will assume default values. > ### - Blank/commented lines are allowed. > ### - Sub-volumes should already be defined above before referring. > > volume shelf-1 > type storage/posix > option directory /glusterfs/shelf-1 > end-volume > > volume shelf-1-readahead > type performance/read-ahead > subvolumes shelf-1 > end-volume > > volume shelf-1-writebehind > type performance/write-behind > option aggregate-size 1MB > option window-size 4MB > subvolumes shelf-1-readahead > end-volume > > ### Add network serving capability to above brick. > volume server > type protocol/server > option transport-type tcp/server # For TCP/IP transport > subvolumes shelf-1-writebehind > option auth.ip.shelf-1-writebehind.allow 127.0.0.1 > end-volume > > > Local client configuration: > > ############################################## > ### GlusterFS Client Volume Specification ## > ############################################## > > volume storage > type protocol/client > option transport-type tcp/client > option remote-host 127.0.0.1 > option remote-subvolume shelf-1-writebehind > end-volume > > Writing directly to the underlying filesystem: > > -bash-3.2# dd if=/dev/zero of=/glusterfs/shelf-1/testing count=20M > > 20971520+0 records in > > 20971520+0 records out > > 10737418240 bytes (11 GB) copied, 230.862 seconds, 46.5 MB/s > > > Writing through glusterfs with no readahead/writebehind: > > -bash-3.2# dd if=/dev/zero of=/storage/testing count=20M > > 20971520+0 records in > > 20971520+0 records out > > 10737418240 bytes (11 GB) copied, 3429.89 seconds, 3.1 MB/s > > > Using the above configuration is even slower: > > -bash-3.2# dd if=/dev/zero of=/storage/testing count=20M > 20971520+0 records in > 20971520+0 records out > 10737418240 bytes (11 GB) copied, 4379.16 seconds, 2.5 MB/s > > > What should I be trying to improve performance? I really need help! > > -- Matt > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users >
Hello! 2008/12/12 Matt Lawrence <matt.lawrence at tamu.edu>:> I am seeing extremely slow performance with glusterfs.[...]> Writing directly to the underlying filesystem: > > -bash-3.2# dd if=/dev/zero of=/glusterfs/shelf-1/testing count=20M[...]> 10737418240 bytes (11 GB) copied, 230.862 seconds, 46.5 MB/s > > > Writing through glusterfs with no readahead/writebehind: > > -bash-3.2# dd if=/dev/zero of=/storage/testing count=20M[...]> 10737418240 bytes (11 GB) copied, 3429.89 seconds, 3.1 MB/s > > > Using the above configuration is even slower: > > -bash-3.2# dd if=/dev/zero of=/storage/testing count=20M[...]> 10737418240 bytes (11 GB) copied, 4379.16 seconds, 2.5 MB/s > > > What should I be trying to improve performance? I really need help!IIRC: - GlusterFS 1.3.x has problems with communication overhead. As a result, transporting small blocks of data is very slow. - (11GB data /20M blocks ~0,5k blocksize) looks small to me I'd try a bigger blocksize with dd to check if that really is the problem. (e.g. -bs=64k or even -bs=1M) If that helps, I'd try writebehind on the client to let GlusterFS transport bigger blocks. http://www.gluster.org/docs/index.php/GlusterFS_Translators_v1.3#Write_Behind_Translator Harald St?rzebecher