samba - Aug 2019 - flood of (auth in progress) connections from unresponsive windows client crashing samba

We have been experiencing a debilitating 'bug' in samba where something
is causing a flood of the messages seen below in smbstatus and the network
drives ( in our case N: ) on all clients become unresponsive.  In fact, the
entire client becomes unresponsive, essentially making them unusable until samba
is restarted.  We first saw this and connected it to the following open bug in
samba (https://bugzilla.samba.org/show_bug.cgi?id=11720).  However, after
mitigating the issue by removing root_squash from the nfs mount, things
stabilized for awhile, but we still see this symptom occur occasionally.  One
reproducible way to trigger it was to try and use the network drive as the user
library install location in R/Rstudio.  After informing all users not to do
that, things seemed to stabilize again, but we are still seeing this occur every
other day or so at seemingly random times.  Most of our users on the clients are
using Stata16 to access data on the mapped drive.

The only additional clue is from the log.clienthostname file and is:
""lookup_name_smbconf for clienthostname$ failed".  This appears
to show the client trying to connect as itself and not a specific user.  I
cannot confirm if this is actually related to the core issue or simply a
coincidence.

We are running samba v4.8.3 on Centos v7.6.1810 and our clients are Windows
Server 2016.

Has anyone experienced something similar in smbstatus?  Do you know of any good
consultants who could help us resolve this crucial issue?  Any help is greatly
appreciated.

Samba version 4.8.3
PID     Username     Group        Machine                                  
Protocol Version  Encryption           Signing
----------------------------------------------------------------------------------------------------------------------------------------
10741   (auth in progress)    redacted (ipv4:redacted)      SMB3_11             
partial(AES-128-CMAC)
10730   user1           G-234    redacted (ipv4:redacted)      SMB3_11          
partial(AES-128-CMAC)
10730  user2            G-234    redacted (ipv4:redacted)      SMB3_11          
partial(AES-128-CMAC)
10752   (auth in progress)     redacted (ipv4:redacted)      SMB3_11            
partial(AES-128-CMAC)
10769   (auth in progress)     redacted (ipv4:redacted)      SMB3_11            
partial(AES-128-CMAC)
10730   user3            G-234    redacted (ipv4:redacted)      SMB3_11         
partial(AES-128-CMAC)
10771   (auth in progress)     redacted (ipv4:redacted)      SMB3_11            
partial(AES-128-CMAC)
10781   (auth in progress)     redacted (ipv4:redacted)      SMB3_11            
partial(AES-128-CMAC)

Thanks,

David Walling

On 30/08/2019 02:52, David Walling via samba wrote:
> We have been experiencing a debilitating 'bug' in samba where
something is causing a flood of the messages seen below in smbstatus and the
network drives ( in our case N: ) on all clients become unresponsive.  In fact,
the entire client becomes unresponsive, essentially making them unusable until
samba is restarted.  We first saw this and connected it to the following open
bug in samba (https://bugzilla.samba.org/show_bug.cgi?id=11720).  However, after
mitigating the issue by removing root_squash from the nfs mount, things
stabilized for awhile, but we still see this symptom occur occasionally.  One
reproducible way to trigger it was to try and use the network drive as the user
library install location in R/Rstudio.  After informing all users not to do
that, things seemed to stabilize again, but we are still seeing this occur every
other day or so at seemingly random times.  Most of our users on the clients are
using Stata16 to access data on the mapped drive.
>
> The only additional clue is from the log.clienthostname file and is:
""lookup_name_smbconf for clienthostname$ failed".  This appears
to show the client trying to connect as itself and not a specific user.  I
cannot confirm if this is actually related to the core issue or simply a
coincidence.
>
> We are running samba v4.8.3 on Centos v7.6.1810 and our clients are Windows
Server 2016.
>
> Has anyone experienced something similar in smbstatus?  Do you know of any
good consultants who could help us resolve this crucial issue?  Any help is
greatly appreciated.
>
> Samba version 4.8.3
> PID     Username     Group        Machine                                  
Protocol Version  Encryption           Signing
>
----------------------------------------------------------------------------------------------------------------------------------------
> 10741   (auth in progress)    redacted (ipv4:redacted)      SMB3_11        
partial(AES-128-CMAC)
> 10730   user1           G-234    redacted (ipv4:redacted)      SMB3_11     
partial(AES-128-CMAC)
> 10730  user2            G-234    redacted (ipv4:redacted)      SMB3_11     
partial(AES-128-CMAC)
> 10752   (auth in progress)     redacted (ipv4:redacted)      SMB3_11       
partial(AES-128-CMAC)
> 10769   (auth in progress)     redacted (ipv4:redacted)      SMB3_11       
partial(AES-128-CMAC)
> 10730   user3            G-234    redacted (ipv4:redacted)      SMB3_11    
partial(AES-128-CMAC)
> 10771   (auth in progress)     redacted (ipv4:redacted)      SMB3_11       
partial(AES-128-CMAC)
> 10781   (auth in progress)     redacted (ipv4:redacted)      SMB3_11       
partial(AES-128-CMAC)
>
> Thanks,
>
> David Walling
Please post your smb.conf

Rowland

I left in some of the parameters I've been testing commented out.

Interestingly, we've noticed another client triggering the same type of
symptoms every morning at around the same time.  Those symptoms being a line
'lookup_name_smbconf for COMPUTERNAME$ failed' and a flood of failed
connection attempts from the same client.  The issue seemed to resolve itself
after a few minutes in this last case.  I happened to be watch smbstatus at the
time it occurred.


[global]
        netbios name = service-samba4
        #socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536
SO_SNDBUF=65536
        #idmap uid = 100000-200000
        #winbind enum users = yes
        winbind gid = 100000-200000
        workgroup = PRODUCTION
        os level = 20
        winbind enum groups = yes
        socket address = dc.production.redacted.org
        password server = *
        preferred master = no
        winbind separator = +
        max log size = 20000
        log level = 1 smbd:10
        log file = /var/log/samba/log.%m
        encrypt passwords = yes
        dns proxy = no
        realm = PRODUCTION.REDACTED.ORG
        security = ADS
        wins server = dc.production.redacted.org
        wins proxy = no

        #oplocks = False
        #level2 oplocks = False

        #dos filemode = yes
        #enable privileges = yes

 username map = /etc/samba/user_and_group_map.txt

        #client max protocol = SMB3_02
        #server max protocol = SMB3_02

        # ACL Settings
        vfs objects = acl_xattr
        map acl inherit = yes
        nt acl support = yes
        store dos attributes = no

        # Multichannel
        #server multi channel support = yes
        aio read size = 0
        aio write size = 0

        # Prevent zombie processes
        deadtime = 15
        csc policy = disable

[share1]
        path = /samba/share1
        browseable = yes
        read only = no
        inherit acls = yes
        inherit permissions = yes
        #oplocks = False
        #level2 oplocks = False
        create mask = 700
        directory mask = 700
        valid users = @"G-817803"
        #acl_xattr:ignore system acl = yes
        hosts allow =  redacted

________________________________
From: Rowland penny <rpenny at samba.org>
Sent: Friday, August 30, 2019 2:52 AM
To: samba at lists.samba.org <samba at lists.samba.org>
Subject: Re: [Samba] flood of (auth in progress) connections from unresponsive
windows client crashing samba

On 30/08/2019 02:52, David Walling via samba wrote:
> We have been experiencing a debilitating 'bug' in samba where
something is causing a flood of the messages seen below in smbstatus and the
network drives ( in our case N: ) on all clients become unresponsive.  In fact,
the entire client becomes unresponsive, essentially making them unusable until
samba is restarted.  We first saw this and connected it to the following open
bug in samba (https://bugzilla.samba.org/show_bug.cgi?id=11720).  However, after
mitigating the issue by removing root_squash from the nfs mount, things
stabilized for awhile, but we still see this symptom occur occasionally.  One
reproducible way to trigger it was to try and use the network drive as the user
library install location in R/Rstudio.  After informing all users not to do
that, things seemed to stabilize again, but we are still seeing this occur every
other day or so at seemingly random times.  Most of our users on the clients are
using Stata16 to access data on the mapped drive.
>
> The only additional clue is from the log.clienthostname file and is:
""lookup_name_smbconf for clienthostname$ failed".  This appears
to show the client trying to connect as itself and not a specific user.  I
cannot confirm if this is actually related to the core issue or simply a
coincidence.
>
> We are running samba v4.8.3 on Centos v7.6.1810 and our clients are Windows
Server 2016.
>
> Has anyone experienced something similar in smbstatus?  Do you know of any
good consultants who could help us resolve this crucial issue?  Any help is
greatly appreciated.
>
> Samba version 4.8.3
> PID     Username     Group        Machine                                  
Protocol Version  Encryption           Signing
>
----------------------------------------------------------------------------------------------------------------------------------------
> 10741   (auth in progress)    redacted (ipv4:redacted)      SMB3_11        
partial(AES-128-CMAC)
> 10730   user1           G-234    redacted (ipv4:redacted)      SMB3_11     
partial(AES-128-CMAC)
> 10730  user2            G-234    redacted (ipv4:redacted)      SMB3_11     
partial(AES-128-CMAC)
> 10752   (auth in progress)     redacted (ipv4:redacted)      SMB3_11       
partial(AES-128-CMAC)
> 10769   (auth in progress)     redacted (ipv4:redacted)      SMB3_11       
partial(AES-128-CMAC)
> 10730   user3            G-234    redacted (ipv4:redacted)      SMB3_11    
partial(AES-128-CMAC)
> 10771   (auth in progress)     redacted (ipv4:redacted)      SMB3_11       
partial(AES-128-CMAC)
> 10781   (auth in progress)     redacted (ipv4:redacted)      SMB3_11       
partial(AES-128-CMAC)
>
> Thanks,
>
> David Walling
Please post your smb.conf

Rowland

/etc/samba/user_and_group_map.txt contains Windows username/group to linux
username/group mappings.  In our setup, all users exist in ldap, as do the
directory groups, but the linux user and group information (namely uid/gid) do
not.  This has been setup such that the users connect to samba as the windows
username (ex. PRODUCTION+user1) for an authroized group (PRODUCTION+group1), but
the files and permissions on the linux samba server are created and managed with
the appropriate uid/gids.

Example:

linuxuser=PRODUCTION+windowsuser
G-234=PRODUCTION+directorygroup

I do not believe we are using sssd, but are using winbind.  Its quite possible
we don't have this setup optimally, but this setup does work as needed,
outside of these occasional crash/unresponsive states.

Thanks!

David W.