samba - Jan 2010 - SMB/CIFS seq. transfers top out at 30MiB/s (NFSv4 and HTTP: 100MiB/s+)

Hello list,

I'm using Samba 3.4.5 on a home-hosted fileserver of mine to easily share 
files with both GNU/Linux and Windows XP (Professional 32Bit SP3) clients. The 
machines are connected to each other via a switched GBit ethernet network, the 
actual available bandwidth between the server and the nodes over TCP amounts 
to about 940-980MBit (according to iperf).

The server's storage backend is aquite potent, and achieves sequential read 
and write speeds well over the network's linespeed (somewhere around
180MiB/s
read and 130MiB/s write at worst and well over 200MiB/s and 160MiB/s, resp., 
at best). When serving files from the very same filesystem as with Samba, 
Apache 2.2 delivers about 110MiB/s on average, and NFSv4-transfers match that 
number in terms of speed.

With Samba and SMB/CIFS, however, the transfer speed tops out at rather 
disappointing 28-30MiB/s, serving both Windows and GNU/Linux clients. I'm 
using the in-kernel CIFS support on the GNU/Linux machines, and the default 
"Attach Network Drive"-feature with Windows. Using the (afaik pure
userspace)
`smbclient` implementation on the GNU machine doesn't change anything to the
better, the speed remains at the aforementioned ~30MiB/s.

The server system isn't really loaded while serving files over CIFS (its
cores
don't even clock to higher frequencies, but remain at a comfortable 1GHz),
and
smbd never consumes substantially more than ~15% CPUtime while reading from 
disk and delivering to the clients.

I tried adopting advice from the Samba manual's performance tuning section 
(http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/speed.html), but 
that didn't improve the situation at all - transfer speed remained roughly
the
same, compared to these settings' default values. Playing with the
clients'
mount options (rsize and wsize, specifically) didn't yield any noticeable 
improvements, either.

I continue to see other peoples' reports on the web that they manage to 
squeeze much higher transfer rates (70MiB/s+) out of dedicated NAS appliances 
via SMB/CIFS (most of which supposedly also run Samba for providing that 
service), and I fail to see why my platform won't deliver similar results,
as
the hardware should easily match whatever those NAS-devices offer.

Below I will list what I think might be relevant information to track down 
what's wrong; in case I'm missing something that'd be of use,
please, let me
know!

----SNIP START: egrep -v '^[[:space:]]*[;#]|^$' /etc/samba/smb.conf ----
[global]
    workgroup = ARBEITSGRUPPE
    security = share
    load printers = no
    guest account = nobody
    dns proxy = no
    syslog = 1
    syslog only = yes
    socket options = SO_RCVBUF=8192 SO_SNDBUF=8192
    disable netbios = yes
    unix extensions = yes
    unix charset = UTF-8
    display charset = UTF-8
    use mmap = yes
    use sendfile = yes
    wins support = no
[files]
   path = /srv/files/pub/
   public = yes
   only guest = yes
   writable = yes
   printable = no
---- SNIP END: egrep -v '^[[:space:]]*[;#]|^$' /etc/samba/smb.conf ----

My GNU/Linux client mounts the share with the following options:
---- SNIP START: /etc/fstab ----
//virtue.local/files /media/network cifs \ 
auto,user,pass=,rw,uid=1000,gid=100,noexec,nolock,\
file_mode=0664,dir_mode=0775,iocharset=utf8,\
wsize=57344,rsize=57344 0 0
---- SNIP END: /etc/fstab ----

The output of `smbd -d` is recorded here: http://pasted.at/fb1889588d_nl.html

All GNU/Linux clients use recent (2.6.32+) kernels and are x86_64 machines. 
All clients are able to get the HTTP transfer speeds described above.

I'm still using the very same kernel that was used to build Samba/smbd on. 
There's nothing out of the ordinary recorded in the server's logs. 
`nmbd`
isn't running on the server, as I don't need that kind of name
resolution
support. Samba is Version 3.4.5, running on Gentoo GNU/Linux ~amd64.


If anyone spots something obvious that might limit transfer speeds in the way 
I described, please leave a comment. Thanks very much in advance for your time 
and effort!
-- 
with best regards:
- Johannes Truschnigg ( johannes at truschnigg.info )

www: http://johannes.truschnigg.info/
phone: +43 650 2 133337
xmpp: johannes at truschnigg.info

Please do not bother me with HTML-eMail or attachments. Thank you.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL:
<http://lists.samba.org/pipermail/samba/attachments/20100120/162ff7c4/attachment.pgp>

On Wed, 20 Jan 2010 19:41:24 +0100
Johannes Truschnigg <johannes at truschnigg.info> wrote:
> Hello list,
> 
> I'm using Samba 3.4.5 on a home-hosted fileserver of mine to easily
share
> files with both GNU/Linux and Windows XP (Professional 32Bit SP3) clients.
The
> machines are connected to each other via a switched GBit ethernet network,
the
> actual available bandwidth between the server and the nodes over TCP
amounts
> to about 940-980MBit (according to iperf).
> 
> The server's storage backend is aquite potent, and achieves sequential
read
> and write speeds well over the network's linespeed (somewhere around
180MiB/s
> read and 130MiB/s write at worst and well over 200MiB/s and 160MiB/s,
resp.,
> at best). When serving files from the very same filesystem as with Samba, 
> Apache 2.2 delivers about 110MiB/s on average, and NFSv4-transfers match
that
> number in terms of speed.
> 
> With Samba and SMB/CIFS, however, the transfer speed tops out at rather 
> disappointing 28-30MiB/s, serving both Windows and GNU/Linux clients.
I'm
> using the in-kernel CIFS support on the GNU/Linux machines, and the default
> "Attach Network Drive"-feature with Windows. Using the (afaik
pure userspace)
> `smbclient` implementation on the GNU machine doesn't change anything
to the
> better, the speed remains at the aforementioned ~30MiB/s.
> 
> The server system isn't really loaded while serving files over CIFS
(its cores
> don't even clock to higher frequencies, but remain at a comfortable
1GHz), and
> smbd never consumes substantially more than ~15% CPUtime while reading from
> disk and delivering to the clients.
> 
> I tried adopting advice from the Samba manual's performance tuning
section
> (http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/speed.html),
but
> that didn't improve the situation at all - transfer speed remained
roughly the
> same, compared to these settings' default values. Playing with the
clients'
> mount options (rsize and wsize, specifically) didn't yield any
noticeable
> improvements, either.
> 
> I continue to see other peoples' reports on the web that they manage to
> squeeze much higher transfer rates (70MiB/s+) out of dedicated NAS
appliances
> via SMB/CIFS (most of which supposedly also run Samba for providing that 
> service), and I fail to see why my platform won't deliver similar
results, as
> the hardware should easily match whatever those NAS-devices offer.
> 
> Below I will list what I think might be relevant information to track down 
> what's wrong; in case I'm missing something that'd be of use,
please, let me
> know!
> 
> ----SNIP START: egrep -v '^[[:space:]]*[;#]|^$' /etc/samba/smb.conf
----
> [global]
>     workgroup = ARBEITSGRUPPE
>     security = share
>     load printers = no
>     guest account = nobody
>     dns proxy = no
>     syslog = 1
>     syslog only = yes
>     socket options = SO_RCVBUF=8192 SO_SNDBUF=8192
>     disable netbios = yes
>     unix extensions = yes
>     unix charset = UTF-8
>     display charset = UTF-8
>     use mmap = yes
>     use sendfile = yes
>     wins support = no
> [files]
>    path = /srv/files/pub/
>    public = yes
>    only guest = yes
>    writable = yes
>    printable = no
> ---- SNIP END: egrep -v '^[[:space:]]*[;#]|^$' /etc/samba/smb.conf
----
> 
> My GNU/Linux client mounts the share with the following options:
> ---- SNIP START: /etc/fstab ----
> //virtue.local/files /media/network cifs \ 
> auto,user,pass=,rw,uid=1000,gid=100,noexec,nolock,\
> file_mode=0664,dir_mode=0775,iocharset=utf8,\
> wsize=57344,rsize=57344 0 0
> ---- SNIP END: /etc/fstab ----
> 
> The output of `smbd -d` is recorded here:
http://pasted.at/fb1889588d_nl.html
> 
> All GNU/Linux clients use recent (2.6.32+) kernels and are x86_64 machines.
> All clients are able to get the HTTP transfer speeds described above.
> 
> I'm still using the very same kernel that was used to build Samba/smbd
on.
> There's nothing out of the ordinary recorded in the server's logs. 
`nmbd`
> isn't running on the server, as I don't need that kind of name
resolution
> support. Samba is Version 3.4.5, running on Gentoo GNU/Linux ~amd64.
> 
> 
> If anyone spots something obvious that might limit transfer speeds in the
way
> I described, please leave a comment. Thanks very much in advance for your
time
> and effort!
Most likely, you're running into the lack of parallelism in Linux' CIFS
client. Writes are done in in turn currently and not in parallel as
they should be.

Fixing it is something we'd like to do, but it's a non-trivial amount
of work and it'll probably be a while before it's fixed.

-- 
Jeff Layton <jlayton at samba.org>

Hi Matt,

thanks very much for your input!

On Thursday 21 January 2010 15:02:09 M D wrote:
> [?]
> socket options =  IPTOS_LOWDELAY TCP_NODELAY
> 
> After this, I'm seeing significantly quicker transfers to windows 7
clients
> - at almost line speed
> [?]
I had those options enabled a while back, and tried once again today - to no 
positive effect whatsoever. Performance has in fact been going down, albeit 
very slightly, with those settings by about 5-10%. I top out at 27MiB/s with 
all my clients.

Thanks for your insights regarding kernel tunables - I already tweaked the 
kernel's TCP parameters (initial window size, rx buffer memory, etc.) to
allow
for better transfer speeds on my GBit link, and other file transfer protocols 
easily hit the speeds I'd like to see with CIFS - that is, more than
100MiB/s.
Right now, even OpenSSH's sftp-server using sftp is faster for me (with 
considerably more load on the transfer's endpoints, of course) than smbd 
serving up CIFS.
-- 
with best regards:
- Johannes Truschnigg ( johannes at truschnigg.info )

www: http://johannes.truschnigg.info/
phone: +43 650 2 133337
xmpp: johannes at truschnigg.info

Please do not bother me with HTML-eMail or attachments. Thank you.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL:
<http://lists.samba.org/pipermail/samba/attachments/20100121/9ce68dbd/attachment.pgp>