Ramiro Magallanes
2010-Mar-26 17:04 UTC
[Gluster-users] Configuration suggestions (aka poor/slow performance on new hardware)
Hello there!
Im working on a 6-nodes cluster, with SuperMicro new hardware.
The cluster have to store a millons of JPG's about (200k-4MB),and little
text files.
Each node is :
-Single Xeon(R) CPU E5405 @ 2.00GHz (4 cores)
-4 GB RAM
-64 bits Distro-based (Debian Lenny)
-3ware 9650 sataII-raid, with 1 logical drive in raid 5 mode, the unit
with 3 sata hardisk of 2TB wdc with 64MB of cache each one.
-Xfs filesystem on each logical unit.
When i run the "genfiles.sh" test on each node in local (in the raid-5
unit) mode i've have the follow results:
-3143 files created in 60 seconds.
and if i comment the "sync" line in the script:
-8947 files created in 60 seconds.
Now , with Gluster mounted (22TB) i run the test and the results are:
-1370 files created in 60 seconds.
Now, I'm running the cluster with standard distributed configuration,
and i was making significant number of change in the test process , but
i obtain the same number of wroted files all the time.
Never more than 1400 files created, and 170mbits of network load (top).
The switching layer is gigabit (obviusly) , and there's no high
resources being used , all is normal.
I'm using the 3.0.3 version of Gluster.
Here is my configuration file (only the last part of the file):
##############################################################################
volume distribute
type cluster/distribute
subvolumes 172.17.15.1-1 172.17.15.2-1 172.17.15.3-1
172.17.15.4-1 172.17.15.5-1 172.17.15.6-1
end-volume
volume writebehind
type performance/write-behind
option cache-size 1MB
option flush-behind on
subvolumes distribute
end-volume
volume readahead
type performance/read-ahead
option page-count 4
subvolumes writebehind
end-volume
volume iocache
type performance/io-cache
option cache-size `grep 'MemTotal' /proc/meminfo | awk
'{print
$2 * 0.2 / 1024}' | cut -f1 -d.`MB
option cache-timeout 1
subvolumes readahead
end-volume
volume iothreads
type performance/io-threads
option thread-count 32 # default is 16
subvolumes distribute
end-volume
volume quickread
type performance/quick-read
option cache-timeout 1
option max-file-size 128kB
subvolumes iocache
end-volume
volume statprefetch
type performance/stat-prefetch
subvolumes quickread
end-volume
##############################################################################
Any idea or suggestion to make the performance goes up?
Thanks everyone!
Stephan von Krawczynski
2010-Mar-26 17:17 UTC
[Gluster-users] Configuration suggestions (aka poor/slow performance on new hardware)
Can you check how things look like when using ext3 instead of xfs? On Fri, 26 Mar 2010 18:04:07 +0100 Ramiro Magallanes <listas at sabueso.org> wrote:> Hello there! > > Im working on a 6-nodes cluster, with SuperMicro new hardware. > The cluster have to store a millons of JPG's about (200k-4MB),and little > text files. > > Each node is : > > -Single Xeon(R) CPU E5405 @ 2.00GHz (4 cores) > -4 GB RAM > -64 bits Distro-based (Debian Lenny) > -3ware 9650 sataII-raid, with 1 logical drive in raid 5 mode, the unit > with 3 sata hardisk of 2TB wdc with 64MB of cache each one. > -Xfs filesystem on each logical unit. > > When i run the "genfiles.sh" test on each node in local (in the raid-5 > unit) mode i've have the follow results: > > -3143 files created in 60 seconds. > > and if i comment the "sync" line in the script: > > -8947 files created in 60 seconds. > > Now , with Gluster mounted (22TB) i run the test and the results are: > > -1370 files created in 60 seconds. > > Now, I'm running the cluster with standard distributed configuration, > and i was making significant number of change in the test process , but > i obtain the same number of wroted files all the time. > Never more than 1400 files created, and 170mbits of network load (top). > > The switching layer is gigabit (obviusly) , and there's no high > resources being used , all is normal. > > I'm using the 3.0.3 version of Gluster. > > Here is my configuration file (only the last part of the file): > > ############################################################################## > volume distribute > type cluster/distribute > subvolumes 172.17.15.1-1 172.17.15.2-1 172.17.15.3-1 > 172.17.15.4-1 172.17.15.5-1 172.17.15.6-1 > end-volume > > volume writebehind > type performance/write-behind > option cache-size 1MB > option flush-behind on > subvolumes distribute > end-volume > > volume readahead > type performance/read-ahead > option page-count 4 > subvolumes writebehind > end-volume > > volume iocache > type performance/io-cache > option cache-size `grep 'MemTotal' /proc/meminfo | awk '{print > $2 * 0.2 / 1024}' | cut -f1 -d.`MB > > option cache-timeout 1 > subvolumes readahead > end-volume > > volume iothreads > type performance/io-threads > option thread-count 32 # default is 16 > subvolumes distribute > end-volume > > volume quickread > type performance/quick-read > option cache-timeout 1 > option max-file-size 128kB > subvolumes iocache > end-volume > > volume statprefetch > type performance/stat-prefetch > subvolumes quickread > end-volume > ############################################################################## > > Any idea or suggestion to make the performance goes up? > Thanks everyone! > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >
Ian Rogers
2010-Mar-26 17:44 UTC
[Gluster-users] Configuration suggestions (aka poor/slow performance on new hardware)
Hi Ramiro ideas off the top of my head: Get rid of performance/quick-read - it has a memory leak bug due to be fixed in gluster v3.0.5 If the files are going to be accessed by a program (which doesn't list the directories often) rather than a user (who might) then you can get rid of performance/stat-prefetch too. In the long term - will this cluster be mostly used for serving files (ie. read-mostly) or will you be creating files as often as reading them? If mostly-read then get rid of performance/write-behind. Also the calculated option cache-size only uses 20% of your ram for the io-cache. Hard code it to a value you know works best. See also my write-up http://www.sirgroane.net/2010/03/tuning-glusterfs-for-apache-on-ec2 perhaps. As you have physical disks then I'm guessing performance/read-ahead should be good for you. Does the genfiles.sh script allow you to simulate multiple processes - if not then you're not seeing the full benefit of your 6 back-end stores... Cheers, Ian On 26/03/2010 17:04, Ramiro Magallanes wrote:> Hello there! > > Im working on a 6-nodes cluster, with SuperMicro new hardware. > The cluster have to store a millons of JPG's about (200k-4MB),and little > text files. > > Each node is : > > -Single Xeon(R) CPU E5405 @ 2.00GHz (4 cores) > -4 GB RAM > -64 bits Distro-based (Debian Lenny) > -3ware 9650 sataII-raid, with 1 logical drive in raid 5 mode, the unit > with 3 sata hardisk of 2TB wdc with 64MB of cache each one. > -Xfs filesystem on each logical unit. > > When i run the "genfiles.sh" test on each node in local (in the raid-5 > unit) mode i've have the follow results: > > -3143 files created in 60 seconds. > > and if i comment the "sync" line in the script: > > -8947 files created in 60 seconds. > > Now , with Gluster mounted (22TB) i run the test and the results are: > > -1370 files created in 60 seconds. > > Now, I'm running the cluster with standard distributed configuration, > and i was making significant number of change in the test process , but > i obtain the same number of wroted files all the time. > Never more than 1400 files created, and 170mbits of network load (top). > > The switching layer is gigabit (obviusly) , and there's no high > resources being used , all is normal. > > I'm using the 3.0.3 version of Gluster. > > Here is my configuration file (only the last part of the file): > > ############################################################################## > volume distribute > type cluster/distribute > subvolumes 172.17.15.1-1 172.17.15.2-1 172.17.15.3-1 > 172.17.15.4-1 172.17.15.5-1 172.17.15.6-1 > end-volume > > volume writebehind > type performance/write-behind > option cache-size 1MB > option flush-behind on > subvolumes distribute > end-volume > > volume readahead > type performance/read-ahead > option page-count 4 > subvolumes writebehind > end-volume > > volume iocache > type performance/io-cache > option cache-size `grep 'MemTotal' /proc/meminfo | awk '{print > $2 * 0.2 / 1024}' | cut -f1 -d.`MB > > option cache-timeout 1 > subvolumes readahead > end-volume > > volume iothreads > type performance/io-threads > option thread-count 32 # default is 16 > subvolumes distribute > end-volume > > volume quickread > type performance/quick-read > option cache-timeout 1 > option max-file-size 128kB > subvolumes iocache > end-volume > > volume statprefetch > type performance/stat-prefetch > subvolumes quickread > end-volume > ############################################################################## > > Any idea or suggestion to make the performance goes up? > Thanks everyone! > > _______________________________________________ > Gluster-users mailing list > Gluster-users at gluster.org > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >-- www.ContactClean.com Making changing email address as easy as clicking a mouse. Helping you keep in touch.