samba - Feb 2017 - Need help troubleshooting TCP thrashing, possible kernel bug?

Very well. Here is the affected smb.conf.
------
[global]
    server min protocol = NT1
    server max protocol = SMB3
    interfaces = 127.0.0.1 10.0.0.8
    bind interfaces only = yes
    encrypt passwords = yes
    dns proxy = no
    strict locking = no
    oplocks = yes
    deadtime = 15
    max log size = 51200
    max open files = 2830016
    logging = file
    load printers = no
    printing = bsd
    printcap name = /dev/null
    disable spoolss = yes
    getwd cache = yes
    guest account = nobody
    map to guest = Bad User
    obey pam restrictions = yes
    directory name cache size = 0
    kernel change notify = no
    panic action = /usr/local/libexec/samba/samba-backtrace
    nsupdate command = /usr/local/bin/samba-nsupdate -g
    server string = backup of files
    ea support = yes
    store dos attributes = yes
    lm announce = yes
    hostname lookups = yes
    unix extensions = no
    acl allow execute always = true
    dos filemode = yes
    multicast dns register = no
    local master = no
    idmap config *: backend = tdb
    idmap config *: range = 10000-90000
    server role = member server
    security = user
    passdb backend = ldapsam:ldap://ldap0.packetdigital.com
    ldap admin dn = cn=admin,dc=packetdigital,dc=com
    ldap suffix = dc=packetdigital,dc=com
    ldap user suffix = ou=Users
    ldap group suffix = ou=Groups
    ldap ssl = off
    ldap replication sleep = 1000
    ldap passwd sync = yes
    ldapsam:trusted = yes
    netbios name = HAMMER
    workgroup = PACKETDIGITAL
    domain logons = yes
    idmap config PACKETDIGITAL: backend = ldap
    idmap config PACKETDIGITAL: range = 10000-90000
    idmap config PACKETDIGITAL: ldap url = ldap0.packetdigital.com
    pid directory = /var/run/samba
    create mask = 0666
    directory mask = 0777
    client ntlmv2 auth = yes
    dos charset = CP437
    unix charset = UTF-8
    log level = 1
    #map unix users to 1 or more names
    ## can map an @group to a username
    #username map = /mnt/storage/configs/samba_users.map
    follow symlinks = yes
    wide links = yes
    unix extensions = no
    create mask = 0660
    idmap uid = 10000-90000
    idmap gid = 10000-90000


[Software]
    path = /mnt/storage/cifs-share/Software
    printable = no
    veto files = /.snapshot/.windows/.mac/.zfs/
    writeable = yes
    browseable = yes
    vfs objects = zfs_space zfsacl
    hide dot files = yes
    guest ok = no
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = true
    zfsacl:acesort = dontcare
    create mask = 0775
    force create mode = 0775
    directory mask = 0775
    force directory mode = 0775
    force group = Software
    valid users = @Software, @Software-RO
    read only = yes
    write list = @Software

------

--
Paul Klapperich

On Wed, Feb 8, 2017 at 4:36 PM, Rowland Penny via samba <
samba at lists.samba.org> wrote:
> On Wed, 8 Feb 2017 15:59:16 -0600
> Paul Klapperich via samba <samba at lists.samba.org> wrote:
>
> > I have a FreeNAS 9.3 server running Samba Version 4.3.6 and a bunch of
> > Windows and Linux clients. Everything's been running fine for a
while
> > and nothing changed on the server.
> >
> > Recently (Jan 27th) some of the Archlinux clients updated from a 4.8.x
> > kernel to a 4.9.x kernel. Again, things ran fine. Then on Jan 30th
> > around 2am the Archlinux clients using 4.9.x kernels and utilizing
> > mount.cifs to access samba shares began thrashing on TCP port 445,
> > causing high CPU load on the server. These machines now cause
> > thrashing after 15-20 minutes whenever a share is mounted using
> > mount.cifs.
> >
> > When it's thrashing, I see thousands of opened ports from a single
> > client: # sockstat -4 | grep 10.0.1.87 | wc
> >    10013   70091  740962
> >
> > And on the client, the port is constant changing:
> > $ netstat -net | grep 10.0.0.8
> > tcp        0      0 10.0.1.87:53122         10.0.0.8:445
> >  ESTABLISHED 0          1253359
> > $ netstat -net | grep 10.0.0.8
> > tcp        0      0 10.0.1.87:53700         10.0.0.8:445
> >  ESTABLISHED 0          1253439
> > $ netstat -net | grep 10.0.0.8
> > tcp        0      0 10.0.1.87:53926         10.0.0.8:445
> >  ESTABLISHED 0          1254557
> > $ netstat -net | grep 10.0.0.8
> > tcp        0      0 10.0.1.87:54148         10.0.0.8:445
> >  ESTABLISHED 0          1253578
> > $ netstat -net | grep 10.0.0.8
> > tcp        0      0 10.0.1.87:54352         10.0.0.8:445
> >  ESTABLISHED 0          1253604
> > $ netstat -net | grep 10.0.0.8
> > tcp        0      0 10.0.1.87:54518         10.0.0.8:445
> >  ESTABLISHED 0          1254685
> > $ netstat -net | grep 10.0.0.8
> > tcp        0      0 10.0.1.87:54698         10.0.0.8:445
> >  ESTABLISHED 0          1252177
> >
> > As a work around, I can downgrade these client machines to any 4.8.x
> > kernel and the issue goes away. My suspicion is something is weird in
> > my smb.conf and a change in the 4.9.x kernels exposes that weirdness.
> > Or maybe there's a bug that was introduced in 4.9 and our setup
> > exposes it.
> >
> > I've built 4.10rc kernels from Linus's git repo and they also
have the
> > problem. The 4.9 kernel I built from Linus's git has the problem,
but
> > the 4.8 kernel I built does not, so I don't think it's related
to any
> > patching done by Archlinux. I don't understand why the issue
didn't
> > happen immediately after upgrading kernels on the 27th, but now it
> > very consistently acts up after less than 20 minutes.
> >
> > Attached is the smb.conf used on one of my FreeNAS servers. I was
> > able to copy that config to an Archlinux system running Samba version
> > 4.5.3 (commenting lines 24, 25, 55, and 79 and adjusting the
> > "interfaces =" line) and the problem persists, so it
doesn't appear
> > to be specific to FreeNas or Samba 4.3.6.
> >
> > --
> > Paul Klapperich
>
> Unfortunately, this list removes attachments, you will need to post
> your smb.conf in the actual message.
>
> Rowland
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>

On Wed, 8 Feb 2017 16:43:53 -0600
Paul Klapperich via samba <samba at lists.samba.org> wrote:
> Very well. Here is the affected smb.conf.
> ------
> [global]
>     server min protocol = NT1
>     server max protocol = SMB3
>     interfaces = 127.0.0.1 10.0.0.8
>     bind interfaces only = yes
>     encrypt passwords = yes
>     dns proxy = no
>     strict locking = no
>     oplocks = yes
>     deadtime = 15
>     max log size = 51200
>     max open files = 2830016
>     logging = file
>     load printers = no
>     printing = bsd
>     printcap name = /dev/null
>     disable spoolss = yes
>     getwd cache = yes
>     guest account = nobody
>     map to guest = Bad User
>     obey pam restrictions = yes
>     directory name cache size = 0
>     kernel change notify = no
>     panic action = /usr/local/libexec/samba/samba-backtrace
>     nsupdate command = /usr/local/bin/samba-nsupdate -g
>     server string = backup of files
>     ea support = yes
>     store dos attributes = yes
>     lm announce = yes
>     hostname lookups = yes
>     unix extensions = no
>     acl allow execute always = true
>     dos filemode = yes
>     multicast dns register = no
>     local master = no
>     idmap config *: backend = tdb
>     idmap config *: range = 10000-90000
>     server role = member server
>     security = user
>     passdb backend = ldapsam:ldap://ldap0.packetdigital.com
>     ldap admin dn = cn=admin,dc=packetdigital,dc=com
>     ldap suffix = dc=packetdigital,dc=com
>     ldap user suffix = ou=Users
>     ldap group suffix = ou=Groups
>     ldap ssl = off
>     ldap replication sleep = 1000
>     ldap passwd sync = yes
>     ldapsam:trusted = yes
>     netbios name = HAMMER
>     workgroup = PACKETDIGITAL
>     domain logons = yes
>     idmap config PACKETDIGITAL: backend = ldap
>     idmap config PACKETDIGITAL: range = 10000-90000
>     idmap config PACKETDIGITAL: ldap url = ldap0.packetdigital.com
>     pid directory = /var/run/samba
>     create mask = 0666
>     directory mask = 0777
>     client ntlmv2 auth = yes
>     dos charset = CP437
>     unix charset = UTF-8
>     log level = 1
>     #map unix users to 1 or more names
>     ## can map an @group to a username
>     #username map = /mnt/storage/configs/samba_users.map
>     follow symlinks = yes
>     wide links = yes
>     unix extensions = no
>     create mask = 0660
>     idmap uid = 10000-90000
>     idmap gid = 10000-90000
> 
> 
> [Software]
>     path = /mnt/storage/cifs-share/Software
>     printable = no
>     veto files = /.snapshot/.windows/.mac/.zfs/
>     writeable = yes
>     browseable = yes
>     vfs objects = zfs_space zfsacl
>     hide dot files = yes
>     guest ok = no
>     nfs4:mode = special
>     nfs4:acedup = merge
>     nfs4:chown = true
>     zfsacl:acesort = dontcare
>     create mask = 0775
>     force create mode = 0775
>     directory mask = 0775
>     force directory mode = 0775
>     force group = Software
>     valid users = @Software, @Software-RO
>     read only = yes
>     write list = @Software
> 
I think you need to go and read 'man smb.conf'

'server role = member server' means it should be joined to a windows
domain, but 'domain logons = yes' makes it a PDC, so which is it ?

Rowland

I think it should be neither. Ugh... FreeNAS UI doesn't let me disable the
Domain Logons setting for some reason, so I'll have to talk to them on that
one and it seems they have no option for me to modify the server role.
Thankfully I can edit smb.conf directly on the archlinux box where I was
able to duplicate the issue.

Setting sever role = auto, domain logons = yes doesn't fix the TCP
thrashing and works with the SIDs already populated in LDAP.

It looks like removing "deadtime = 15" from the smb.conf prevents the
TCP
thrashing. If I set "deadtime  = 1" then the thrashing happens after 3
minutes or so.

I'll see if I can put together a conf file that isn't a disaster but
still
expresses the error.

--
Paul Klapperich

On Wed, Feb 8, 2017 at 5:05 PM, Rowland Penny via samba <
samba at lists.samba.org> wrote:
> On Wed, 8 Feb 2017 16:43:53 -0600
> Paul Klapperich via samba <samba at lists.samba.org> wrote:
>
> > Very well. Here is the affected smb.conf.
> > ------
> > [global]
> >     server min protocol = NT1
> >     server max protocol = SMB3
> >     interfaces = 127.0.0.1 10.0.0.8
> >     bind interfaces only = yes
> >     encrypt passwords = yes
> >     dns proxy = no
> >     strict locking = no
> >     oplocks = yes
> >     deadtime = 15
> >     max log size = 51200
> >     max open files = 2830016
> >     logging = file
> >     load printers = no
> >     printing = bsd
> >     printcap name = /dev/null
> >     disable spoolss = yes
> >     getwd cache = yes
> >     guest account = nobody
> >     map to guest = Bad User
> >     obey pam restrictions = yes
> >     directory name cache size = 0
> >     kernel change notify = no
> >     panic action = /usr/local/libexec/samba/samba-backtrace
> >     nsupdate command = /usr/local/bin/samba-nsupdate -g
> >     server string = backup of files
> >     ea support = yes
> >     store dos attributes = yes
> >     lm announce = yes
> >     hostname lookups = yes
> >     unix extensions = no
> >     acl allow execute always = true
> >     dos filemode = yes
> >     multicast dns register = no
> >     local master = no
> >     idmap config *: backend = tdb
> >     idmap config *: range = 10000-90000
> >     server role = member server
> >     security = user
> >     passdb backend = ldapsam:ldap://ldap0.packetdigital.com
> >     ldap admin dn = cn=admin,dc=packetdigital,dc=com
> >     ldap suffix = dc=packetdigital,dc=com
> >     ldap user suffix = ou=Users
> >     ldap group suffix = ou=Groups
> >     ldap ssl = off
> >     ldap replication sleep = 1000
> >     ldap passwd sync = yes
> >     ldapsam:trusted = yes
> >     netbios name = HAMMER
> >     workgroup = PACKETDIGITAL
> >     domain logons = yes
> >     idmap config PACKETDIGITAL: backend = ldap
> >     idmap config PACKETDIGITAL: range = 10000-90000
> >     idmap config PACKETDIGITAL: ldap url = ldap0.packetdigital.com
> >     pid directory = /var/run/samba
> >     create mask = 0666
> >     directory mask = 0777
> >     client ntlmv2 auth = yes
> >     dos charset = CP437
> >     unix charset = UTF-8
> >     log level = 1
> >     #map unix users to 1 or more names
> >     ## can map an @group to a username
> >     #username map = /mnt/storage/configs/samba_users.map
> >     follow symlinks = yes
> >     wide links = yes
> >     unix extensions = no
> >     create mask = 0660
> >     idmap uid = 10000-90000
> >     idmap gid = 10000-90000
> >
> >
> > [Software]
> >     path = /mnt/storage/cifs-share/Software
> >     printable = no
> >     veto files = /.snapshot/.windows/.mac/.zfs/
> >     writeable = yes
> >     browseable = yes
> >     vfs objects = zfs_space zfsacl
> >     hide dot files = yes
> >     guest ok = no
> >     nfs4:mode = special
> >     nfs4:acedup = merge
> >     nfs4:chown = true
> >     zfsacl:acesort = dontcare
> >     create mask = 0775
> >     force create mode = 0775
> >     directory mask = 0775
> >     force directory mode = 0775
> >     force group = Software
> >     valid users = @Software, @Software-RO
> >     read only = yes
> >     write list = @Software
> >
>
> I think you need to go and read 'man smb.conf'
>
> 'server role = member server' means it should be joined to a
windows
> domain, but 'domain logons = yes' makes it a PDC, so which is it ?
>
> Rowland
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>