samba - Mar 2012 - This old tune again : read performance optimisations

Hi list,
I have this problem of (relatively) poor samba read performance. Read
performance is constantly, reproducibly lower than write performance
and I can't really understand why.

Here is the server configuration :

dual octocore Opteron 6128, 32 GB RAM
Adaptec 6445 RAID card
36x 2TB Hitachi SATA drives (raid 60)

Sustained local disk throughput: over 850 MB/s write, over 1.4 GB/s
read.

kernel 3.1.10 (but I tried various versions with no more
success), amd64, pure vanilla kernel.org version.
Distro Debian squeeze amd64, samba  2:3.5.6~dfsg-3squeeze5

The client is an opteron dual core machine running Win7 64 bits. This is
by far the best performing CIFS client I've tested, software wise (see
below). 

The systems are connected in 10GigE ethernet using Myricom Myri-10G
dual ports (only one port used) through a Fujitsu 10GigE switch.

I first tested the same client to the same server using Linux (same
configuration as the server) and  netperf, then NFS, then cifs mount.

Netperf (TCP_SENDFILE test) gives more than 9500 Mb/s both direction, so
the network hardware works OK.

NFS gives the following results both in UDP and TCP modes:
Sustained throughput: 490 to 520 MB/s write, 620 to 650 MB/s read. 

The linux CIFS client running on the same machine writes at a puny 180
MB/s and read at a miserable 75 MB/s. OK, never mind, samba is for
windows clients, right?

Now, the same machine rebooted under windows 7 writes at 450 MB/s,
which is close enough to NFS performance, but can't top 400 MB/s
reading. Read performance *should* be higher. I've played with various
parameters; I can get better write performance at times, on par with
NFS, but I never at any time remotely get as good a read performance.

I'd really like to get this beast tamed at last :)


Here comes the smb.conf file :

[global]
    name resolve order = wins lmhosts host bcast
    socket options = TCP_NODELAY
    
    interfaces = eth2
    bind interfaces only = yes
    aio read size = 1
    aio write size = 1
    aio write behind = yes
 
    passwd chat = *Enter\snew\sUNIX\spassword:* %n\n
*Retype\snew\sUNIX\spassword:* %n\n . encrypt passwords = true
    passdb backend = tdbsam
    unix password sync = false
    passwd program = /usr/bin/passwd %u
    
    winbind separator = +
    winbind enum users = yes
    winbind enum groups = yes
    winbind cache time = 10
    idmap gid = 10000-20000
    idmap uid = 10000-20000
    obey pam restrictions = yes
    
    preserve case = yes
    short preserve case = yes
    inherit acls = yes
    nt acl support = yes
    dns proxy = no
    inherit permissions = yes
    
    load printers = no
    printcap name = /dev/null 
    disable spoolss = yes 
    printing = cups

    invalid users = root

    local master = no
    domain master = no
    preferred master = no
    workgroup = WORKGROUP
    syslog only = no
    os level = 20
    security = user
    max log size = 1000
    load printers = no
    
    guest account = nobody
    # wins support = no
    # wins server = 127.0.0.1
    template shell = /bin/false
    server string = %h server (Samba %v)
    syslog = 0;
    panic action = /usr/share/samba/panic-action %d
    block size = 4096


[nas]
    writeable = yes
    public = yes
    path = /mnt/raid

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac at intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

On Thu, Mar 22, 2012 at 05:49:44PM +0100, Emmanuel Florac
wrote:
> Hi list,
> I have this problem of (relatively) poor samba read performance. Read
> performance is constantly, reproducibly lower than write performance
> and I can't really understand why.
> 
> Here is the server configuration :
> 
> dual octocore Opteron 6128, 32 GB RAM
> Adaptec 6445 RAID card
> 36x 2TB Hitachi SATA drives (raid 60)
> 
> Sustained local disk throughput: over 850 MB/s write, over 1.4 GB/s
> read.
> 
> kernel 3.1.10 (but I tried various versions with no more
> success), amd64, pure vanilla kernel.org version.
> Distro Debian squeeze amd64, samba  2:3.5.6~dfsg-3squeeze5
> 
> The client is an opteron dual core machine running Win7 64 bits. This is
> by far the best performing CIFS client I've tested, software wise (see
> below). 
> 
> The systems are connected in 10GigE ethernet using Myricom Myri-10G
> dual ports (only one port used) through a Fujitsu 10GigE switch.
> 
> I first tested the same client to the same server using Linux (same
> configuration as the server) and  netperf, then NFS, then cifs mount.
> 
> Netperf (TCP_SENDFILE test) gives more than 9500 Mb/s both direction, so
> the network hardware works OK.
> 
> NFS gives the following results both in UDP and TCP modes:
> Sustained throughput: 490 to 520 MB/s write, 620 to 650 MB/s read. 
> 
> The linux CIFS client running on the same machine writes at a puny 180
> MB/s and read at a miserable 75 MB/s. OK, never mind, samba is for
> windows clients, right?
> 
> Now, the same machine rebooted under windows 7 writes at 450 MB/s,
> which is close enough to NFS performance, but can't top 400 MB/s
> reading. Read performance *should* be higher. I've played with various
> parameters; I can get better write performance at times, on par with
> NFS, but I never at any time remotely get as good a read performance.
> 
> I'd really like to get this beast tamed at last :)
> 
> 
> Here comes the smb.conf file :
> 
> [global]
>     name resolve order = wins lmhosts host bcast
>     socket options = TCP_NODELAY
>     
>     interfaces = eth2
>     bind interfaces only = yes
>     aio read size = 1
>     aio write size = 1
>     aio write behind = yes
Until the next 3.6.x ships with the aio_pthread module,
you might we better off removing the "aio XXX" lines.

Linux glibc aio is fundamentally broken (and Ulrich
doesn't seem interested in fixing it) - this is why
I ended up writing the aio_pthread module.

You could try using Volker's aio_fork module, which
might help (with the Linux clients, but probably not
the Windows clients unless you're running SMB2).

You're you're more adventurous you could try running
from v3-6-test git tree and ensure you have the
aio_pthread module loaded.

Jeremy.