samba - Jun 2014 - Troubleshooting poor (small) random read performance -- serverid.tdb?

Hi everyone;

Have a Windows 2012 based workload that generates many (20K+ PPS) small
reads (Wireshark tells me the individual packets are typically less
than 200 bytes and are Read AndX Request) from eight or so hosts to a
Samba server running on RHEL 6.5.  Samba version is the latest Red Hat
provided 3.6.9 version.

Things are going quite a bit slower than expected (as compared to the
same workload pointed at a Windows 2012 server).

iostat shows not much disk IO activity or iowait going on, but smbd
processes are all maxed out CPU wise.  An strace on them shows *lots*
of activity (almost exclusively) to the serverid.tdb file:

fcntl(16, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=636, len=1}) = 0
fcntl(16, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=636, len=1}) = 0

File handle 16 corresponds with the serverid.tdb file for the PID in
question.

Have been searching around trying to find out what exactly it is that
the serverid.tdb file is used for but hvaen't found a great
explanation.  Am wondering if it has something to do with my use of
security ads and winbind?

I'm hoping that if I can find a way to eliminate or optimize all of
this activity I can get better performance out of Samba and avoid
needing to shift to Windows.

Possibly Samba 4.x would work better?  Haven't yet tried.

Also am unsure if this random read workload with very small
transactions will work well out of the box with Samba.

My config:

[global]
    workgroup = WORKGROUP
    password server = server, *
    realm = realm.com
    security = ads
    idmap uid = 10000-19999
    idmap gid = 10000-19999
    idmap config WORKGROUP:backend = rid
    idmap config WORKGROUP:range = 10000000-19999999
    template shell = /bin/bash
    winbind enum users = no
    winbind enum groups = no
    winbind separator = +
    winbind use default domain = yes
    winbind normalize names = yes
    template homedir = /home/%D/%U
    template shell = /bin/bash
    server string = Samba Server Version %v
    log file = /var/log/samba/log.%m
    log level= 1
    max log size = 50
    passdb backend = tdbsam
    load printers = no
    cups options = raw

    socket options = TCP_NODELAY SO_KEEPALIVE SO_RCVBUF=131072 SO_SNDBUF=131072
IPTOS_LOWDELAY
    # These are defaults, but explicitly set here.
    read raw = yes
    write raw = yes
    strict allocate = no
    # Unsure is this is valid?
    min receivefile size = 16384


Thanks,
Ray

On Tue, Jun 17, 2014 at 10:27:38PM -0700, Ray Van Dolson
wrote:
> Hi everyone;
> 
> Have a Windows 2012 based workload that generates many (20K+ PPS) small
> reads (Wireshark tells me the individual packets are typically less
> than 200 bytes and are Read AndX Request) from eight or so hosts to a
> Samba server running on RHEL 6.5.  Samba version is the latest Red Hat
> provided 3.6.9 version.
> 
> Things are going quite a bit slower than expected (as compared to the
> same workload pointed at a Windows 2012 server).
> 
> iostat shows not much disk IO activity or iowait going on, but smbd
> processes are all maxed out CPU wise.  An strace on them shows *lots*
> of activity (almost exclusively) to the serverid.tdb file:
> 
> fcntl(16, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=636, len=1}) = 0
> fcntl(16, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=636, len=1}) = 0
> 
> File handle 16 corresponds with the serverid.tdb file for the PID in
> question.
A lot of accesses to serverid.tdb have been eliminated in
Samba 4.1. In Samba master, we eliminated the requirement to
do fcntls altogether by introducing mutex support to tdb.
> 
> Have been searching around trying to find out what exactly it is that
> the serverid.tdb file is used for but hvaen't found a great
> explanation.  Am wondering if it has something to do with my use of
> security ads and winbind?
No, that's not the cause.
> I'm hoping that if I can find a way to eliminate or optimize all of
> this activity I can get better performance out of Samba and avoid
> needing to shift to Windows.
If your workload does not do byte range locks and does not
depend on those, you might give "strict locking = no" a
quick try.
> Possibly Samba 4.x would work better?  Haven't yet tried.
Hopefully 4.1 would be better. Also, depending on the
workload, enabling async I/O (aio read size = 1, aio write
size = 1) might also improve things significantly.
> Also am unsure if this random read workload with very small
> transactions will work well out of the box with Samba.
> 
> My config:
> 
> [global]
>     workgroup = WORKGROUP
>     password server = server, *
>     realm = realm.com
>     security = ads
>     idmap uid = 10000-19999
>     idmap gid = 10000-19999
>     idmap config WORKGROUP:backend = rid
>     idmap config WORKGROUP:range = 10000000-19999999
>     template shell = /bin/bash
>     winbind enum users = no
>     winbind enum groups = no
>     winbind separator = +
>     winbind use default domain = yes
>     winbind normalize names = yes
>     template homedir = /home/%D/%U
>     template shell = /bin/bash
>     server string = Samba Server Version %v
>     log file = /var/log/samba/log.%m
>     log level= 1
>     max log size = 50
>     passdb backend = tdbsam
>     load printers = no
>     cups options = raw
> 
>     socket options = TCP_NODELAY SO_KEEPALIVE SO_RCVBUF=131072
SO_SNDBUF=131072 IPTOS_LOWDELAY
Please remove these settings if you are not running ancient
AIX or so. Modern kernels are good at figuring out those
settings themselves. In particular the SNDBUV/RCVBUF
settings can cause harm.

If you are in the position to try 4.1, I would suggest to do
so. 4.1 also has proper support for SMB2, which might also
have a huge effect on overall performance.

Please keep us posted about your results. If nothing helps,
don't give up but provide us with more input please :-)

Volker

-- 
SerNet GmbH, Bahnhofsallee 1b, 37081 G?ttingen
phone: +49-551-370000-0, fax: +49-551-370000-9
AG G?ttingen, HRB 2816, GF: Dr. Johannes Loxen
http://www.sernet.de, mailto:kontakt at sernet.de