Gluster users - Oct 2012 - low samba performance with glusterfs backend

Hello folks,

We test samba performance with local ext4 and glusterfs backends, it shows
performance is very different.
The samba server has 4 1Gbps NICs and bond with mode 6, backend storage is raid0
with 12 SAS disks.
A LUN is created over all disks, make as EXT4 file system, and used as glusterfs
brick.

On the samba server, use dd test local ext4 and glusterfs, write bandwidth are
477MB/s and 357MB/s, details as follow.
When mount cifs on Centos 6.0, write bandwidth downgrade to 184MB/s and 117MB/s,
very different.
Using 4 win7 clients to test with SANergy/Iometer, cifs over ext4 aggregate
throughput can get about 391 MB/s.
However, cifs over glusterfs aggregate throughput only can get about 180 MB/s.

Why CIFS performance is so slow over glusterfs backend? 
Anybody meet such issues and how to resolve(samba and glusterfs parameters)? 
thanks a lot in advance.

#test local ext4
[root at pana53 5f4554cf-3b56-43e5-847e-cb409b0edc30]# dd if=/dev/zero of=dd.dat
bs=1MB count=10k
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 21.4764 s, 477 MB/s

#test local glusterfs
[root at pana53 61659a17-69bd-4704-81fe-0853f1c891cd]# dd if=/dev/zero of=dd.dat
bs=1MB count=10k
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 28.7148 s, 357 MB/s

#test cifs over ext4
[root at pana53 ext4-share]# dd if=/dev/zero of=dd.dat bs=1MB count=10k
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 55.5549 s, 184 MB/s

#test cifs over glusterfs
[root at pana53 glusterfs-share]# dd if=/dev/zero of=dd.dat bs=1MB count=10k
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 87.4899 s, 117 MB/s


here is smb.conf:
[global]
        server string = samba server
        map to guest = Bad Password
        log file = /var/log/samba/log.%m
        max log size = 50
        max protocol = SMB2
        socket options = TCP_NODELAY IPTOS_LOWDELAY
        idmap config * : backend = tdb
        aio read size = 16384
        aio write size = 16384
        use sendfile = Yes
        posix locking = No

[ext4-share]
        comment = None
        path = /data/5f4554cf-3b56-43e5-847e-cb409b0edc30
        read only = No
        guest ok = Yes

[glusterfs-share]
        comment = None
        path = /reexport/61659a17-69bd-4704-81fe-0853f1c891cd
        read only = No
        guest ok = Yes 

here is gluster volume info:
[root at pana53 glusterfs-share]# gluster volume info 
Volume Name: vol1
Type: Distribute
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: pana53:/data/5f4554cf-3b56-43e5-847e-cb409b0edc30
Options Reconfigured:
auth.allow: *,192.168.1.*
features.quota: on
nfs.disable: on

BR,
liuben



nuaa_liuben at sina.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121013/507da560/attachment.html>

On 10/13/2012 04:45 AM, nuaa_liuben wrote:
> here is smb.conf:
> [global]
>         server string = samba server
>         map to guest = Bad Password
>         log file = /var/log/samba/log.%m
>         max log size = 50
>         max protocol = SMB2
>         socket options = TCP_NODELAY IPTOS_LOWDELAY
>         idmap config * : backend = tdb
>         aio read size = 16384
>         aio write size = 16384
>         use sendfile = Yes
>         posix locking = No
hi,

We use this configuration:

[global]
     socket options = TCP_NODELAY IPTOS_LOWDELAY SO_SNDBUF=131072 
SO_RCVBUF=131072
     read raw = yes
     server string = %h
     write raw = yes
     #oplocks = yes
     max xmit = 131072
     dead time = 15
     getwd cache = yes
     use sendfile=yes
     block size = 131072
     load printers = no
     aio write size = 16384
     aio write behind = true
     wins support = no
     local master = no


It works with the best performance we could achive, though I don't 
remember the exact numbers.
Maybe you can get some tips.

tamas

ps.: Please don't do crossposting.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121013/ee5aa8a7/attachment.html>