Gluster users - Aug 2013 - Slow on writing

Hello, guys


I wrote small script :
#!/bin/bash

for i in {1..1000}; do
size=$((RANDOM%5+1))
dd if=/dev/zero of=/storage/test/bigfile${i} count=1024 bs=${size}k
done

This script creates files with different size on volume

here is output:
2097152 bytes (2.1 MB) copied, 0.120632 s, 17.4 MB/s
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.14548 s, 7.2 MB/s
1024+0 records in
1024+0 records out
2097152 bytes (2.1 MB) copied, 0.125532 s, 16.7 MB/s
1024+0 records in
1024+0 records out
3145728 bytes (3.1 MB) copied, 0.144503 s, 21.8 MB/s
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.0994717 s, 10.5 MB/s
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 0.142613 s, 29.4 MB/s
1024+0 records in
1024+0 records out
4194304 bytes (4.2 MB) copied, 0.103823 s, 40.4 MB/s
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.138864 s, 7.6 MB/s
1024+0 records in
1024+0 records out
3145728 bytes (3.1 MB) copied, 0.102374 s, 30.7 MB/s
1024+0 records in
1024+0 records out
3145728 bytes (3.1 MB) copied, 0.166409 s, 18.9 MB/s
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.169923 s, 6.2 MB/s
1024+0 records in
1024+0 records out
2097152 bytes (2.1 MB) copied, 0.142017 s, 14.8 MB/s
1024+0 records in
1024+0 records out
2097152 bytes (2.1 MB) copied, 0.159753 s, 13.1 MB/s
1024+0 records in
1024+0 records out
3145728 bytes (3.1 MB) copied, 0.146142 s, 21.5 MB/s
^C180+0 records in
180+0 records out
737280 bytes (737 kB) copied, 0.0306554 s, 24.1 MB/s


as you can see - the speed very slow :(

I  have  configured ethernet networks on bricks - as bond (but only on
ope peer)
Output of iperf
 iperf -c 192.168.15.165
------------------------------------------------------------
Client connecting to 192.168.15.165, TCP port 5001
TCP window size:  640 KByte (default)
------------------------------------------------------------
[  3] local 192.168.15.159 port 37095 connected with 192.168.15.165 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.03 GBytes   880 Mbits/sec


I  will  run this command dd if=/dev/zero of=/storage/test/bigfile${i}
count=1024 bs=10k on peer.. speed is very high:

root at nas:~#  dd if=/dev/zero of=/storage/test/bigfile${i} count=1024 bs=10k
1024+0 records in
1024+0 records out
10485760 bytes (10 MB) copied, 0.0114412 s, 916 MB/s
root at nas:~#

my volume config:
root at nas:~# gluster volume info

Volume Name: storage
Type: Replicate
Volume ID: 8abee05f-9aa1-41d7-9f72-363c6fd8fc74
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: nas.storage:/storage
Brick2: back.storage:/storage
Options Reconfigured:
performance.cache-size: 256MB


my  installation  is  default, any one have good guide how to get good
performance ?

Thank you


-------------------------------------------------------
??????? ????????? ?????????????
??????? ?????
???? "?????? kg" - http://www.hoster.kg
??. ????????? 123 (?????? ????)
help at hoster.kg

of example same client wrote file to nfs share
root at ispcp:/mnt# dd if=/dev/zero of=./bigfile${i} count=1024 bs=10k
1024+0 records in
1024+0 records out
10485760 bytes (10 MB) copied, 0.133489 s, 78.6 MB/s

much faster :(

cat /etc/mtab

fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
nas.storage:/storage /storage fuse.glusterfs
rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
0 0

how to mount /storage on client with noatime, nodiratime ?

on peers:
1) RAID 10 (hardware)
/dev/sda1 /storage ext4
rw,noatime,nodiratime,errors=remount-ro,user_xattr,noacl,barrier=1,data=ordered
0 0
2) RAID 5 (software)
/dev/md5 /storage ext4 rw,noatime,nodiratime,noacl 0 0




-------------------------------------------------------
??????? ????????? ?????????????
??????? ?????
???? "?????? kg" - http://www.hoster.kg
??. ????????? 123 (?????? ????)
help at hoster.kg

On 08/16/2013 11:21 PM, Alexey Shalin wrote:
> I wrote small script :
> #!/bin/bash
>
> for i in {1..1000}; do
> size=$((RANDOM%5+1))
> dd if=/dev/zero of=/storage/test/bigfile${i} count=1024 bs=${size}k
> done
>
> This script creates files with different size on volume
>
> here is output:
> 2097152 bytes (2.1 MB) copied, 0.120632 s, 17.4 MB/s
> 1024+0 records in
> 1024+0 records out
> 1048576 bytes (1.0 MB) copied, 0.14548 s, 7.2 MB/s
> 1024+0 records in
> 1024+0 records out
It looks like you're doing small writes (1-6KB) from a single thread.  That 
means network latency is going to be your primary limiting factor.  20MB/s at 
4KB is 5000 IOPS or 0.2ms per network round trip.  You don't say what kind
of
network you're using, but if it's Plain Old GigE that doesn't seem
too
surprising.  BTW, the NFS numbers are likely to be better because the NFS 
client does more caching and you're not writing enough to fill memory, so 
you're actually getting less durability than in the native-protocol test and
therefore the numbers aren't directly comparable.

I suggest trying larger block sizes and higher I/O thread counts (with iozone 
you can do this in a single command instead of a script).  You should see a 
pretty marked improvement.