nuaa_liuben
2012-Oct-13 02:45 UTC
[Gluster-users] low samba performance with glusterfs backend
Hello folks,
We test samba performance with local ext4 and glusterfs backends, it shows
performance is very different.
The samba server has 4 1Gbps NICs and bond with mode 6, backend storage is raid0
with 12 SAS disks.
A LUN is created over all disks, make as EXT4 file system, and used as glusterfs
brick.
On the samba server, use dd test local ext4 and glusterfs, write bandwidth are
477MB/s and 357MB/s, details as follow.
When mount cifs on Centos 6.0, write bandwidth downgrade to 184MB/s and 117MB/s,
very different.
Using 4 win7 clients to test with SANergy/Iometer, cifs over ext4 aggregate
throughput can get about 391 MB/s.
However, cifs over glusterfs aggregate throughput only can get about 180 MB/s.
Why CIFS performance is so slow over glusterfs backend?
Anybody meet such issues and how to resolve(samba and glusterfs parameters)?
thanks a lot in advance.
#test local ext4
[root at pana53 5f4554cf-3b56-43e5-847e-cb409b0edc30]# dd if=/dev/zero of=dd.dat
bs=1MB count=10k
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 21.4764 s, 477 MB/s
#test local glusterfs
[root at pana53 61659a17-69bd-4704-81fe-0853f1c891cd]# dd if=/dev/zero of=dd.dat
bs=1MB count=10k
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 28.7148 s, 357 MB/s
#test cifs over ext4
[root at pana53 ext4-share]# dd if=/dev/zero of=dd.dat bs=1MB count=10k
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 55.5549 s, 184 MB/s
#test cifs over glusterfs
[root at pana53 glusterfs-share]# dd if=/dev/zero of=dd.dat bs=1MB count=10k
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 87.4899 s, 117 MB/s
here is smb.conf:
[global]
server string = samba server
map to guest = Bad Password
log file = /var/log/samba/log.%m
max log size = 50
max protocol = SMB2
socket options = TCP_NODELAY IPTOS_LOWDELAY
idmap config * : backend = tdb
aio read size = 16384
aio write size = 16384
use sendfile = Yes
posix locking = No
[ext4-share]
comment = None
path = /data/5f4554cf-3b56-43e5-847e-cb409b0edc30
read only = No
guest ok = Yes
[glusterfs-share]
comment = None
path = /reexport/61659a17-69bd-4704-81fe-0853f1c891cd
read only = No
guest ok = Yes
here is gluster volume info:
[root at pana53 glusterfs-share]# gluster volume info
Volume Name: vol1
Type: Distribute
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: pana53:/data/5f4554cf-3b56-43e5-847e-cb409b0edc30
Options Reconfigured:
auth.allow: *,192.168.1.*
features.quota: on
nfs.disable: on
BR,
liuben
nuaa_liuben at sina.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121013/507da560/attachment.html>
Papp Tamas
2012-Oct-13 21:02 UTC
[Gluster-users] low samba performance with glusterfs backend
On 10/13/2012 04:45 AM, nuaa_liuben wrote:> here is smb.conf: > [global] > server string = samba server > map to guest = Bad Password > log file = /var/log/samba/log.%m > max log size = 50 > max protocol = SMB2 > socket options = TCP_NODELAY IPTOS_LOWDELAY > idmap config * : backend = tdb > aio read size = 16384 > aio write size = 16384 > use sendfile = Yes > posix locking = Nohi, We use this configuration: [global] socket options = TCP_NODELAY IPTOS_LOWDELAY SO_SNDBUF=131072 SO_RCVBUF=131072 read raw = yes server string = %h write raw = yes #oplocks = yes max xmit = 131072 dead time = 15 getwd cache = yes use sendfile=yes block size = 131072 load printers = no aio write size = 16384 aio write behind = true wins support = no local master = no It works with the best performance we could achive, though I don't remember the exact numbers. Maybe you can get some tips. tamas ps.: Please don't do crossposting. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121013/ee5aa8a7/attachment.html>