Gluster users - Nov 2009 - very slow replicated performance on parallel fs calls

Hello,
We are trying to configure gluster replication for a cyrus imap email server.
Even after heavy tweaking configs for small files, (bug 314 was solved at 2.0.8
but i did not backported it to debian lenny yet), the performance for parallel
executions over the fs are VERY painfully slow, for a 3 mirrors set.
We used, instead of the cyrus server, for preliminary simple tests, the postmark
sw and some shell scripting, before advancing for bonnie++. The cyrus behaviour
fs use is "conceptually similar".
The parallel shell execution:
http://techie-mee.blogspot.com/2008/05/run-shell-commands-in-parallel.html
It is a slow shell script. But enough for some preliminary comparison tests.

Please, read below. I removed the long verbose parallel log output. Take a look
at times and quantities.
The script for removing long list of files is simple. Only a loop of rm by line
output of ls.
Read the time difference for removing files sequentially or in parallel calls.
Also, for comparison, see at an ext3 local directory. The parallel shell script
is slow, but performs much faster at local ext3 directory.
This very poor speed behaviour is seen as the server spawns many requests to the
filesystem also.

What could be done?
Any suggestions?
Regards.
Andre Felipe Machado
http://www.techforce.com.br

debian459140:~# mount -t glusterfs -o suid,noatime
/root/arquivos_glusterfs/testes/glusterfs.vol.afr.tcpnodelay.noreadahead.statprefetch.laboratorio
/mnt/glusterfs

debian459140:/mnt/glusterfs# yes | head -c 100K | split -b 1 -a 6
debian459140:~# ls /mnt/glusterfs | wc -l
102400
debian459140:~# find /mnt/glusterfs -type f | sed -e 's/\(.*\)/rm \1/g'
> tmp.txt

debian459140:~# time cat tmp.txt | sh parallel.sh 8
aborted due slow speed

real	14m21.131s
user	0m0.784s
sys	0m1.008s
debian459140:/mnt/glusterfs# ls /mnt/glusterfs |wc -l
102239
debian459140:/mnt/glusterfs# 
debian459140:/mnt/glusterfs# time sh
/root/arquivos_glusterfs/testes/remove_millions_files_mnt_glusterfs.sh 

real	7m37.446s
user	0m18.681s
sys	1m43.238s
debian459140:/mnt/glusterfs# 

yes | head -c 100K | split -b 1 -a 6

debian459140:~# find /root/temp -type f | sed -e 's/\(.*\)/rm \1/g' >
tmp.txt
debian459140:~# tail tmp.txt
rm /root/temp/xaafoel
rm /root/temp/xaafnuv
rm /root/temp/xaabjzy
rm /root/temp/xaaeggu
rm /root/temp/xaaeyqk
rm /root/temp/xaabque
rm /root/temp/xaabkhg
rm /root/temp/xaafrsp
rm /root/temp/xaaerzj
rm /root/temp/xaadgom
debian459140:~# time cat tmp.txt | sh parallel.sh 8
real	62m26.245s
user	73m17.499s
sys	45m54.600s
debian459140:~#

Hello,
The solution is to use the server performance translator io-threads on the
CLIENT side TOO.
I used it at very first translator order (is it the best position?) and
performance for PARALLEL filesystem operations improved a LOT, giving near
performance of serial operations (that still needs to improve for small files).
Regards.
Andre Felipe Machado
http://www.techforce.com.br