samba - Oct 2017 - System load problem with samba 4.4.2 caused by many ntlm auth client requests

Hello,

since a while I experience a strange problem with my samba 4.4.2 running
on a SLES12SP2 system. The server does what it is supposed to do, so
users can work without any problems and access their files via smb but
since some weeks the server  shows a strange and unusual very high
system load.

The samba server is not the domaincrontroller (which is a windows
machine)  but member of the domain and offers all windows clients access
to shares stored on linux file servers.

When watching processes on the samba server via top I see a system load
with a minimum of 18 and up to 50 (the server VM has 6 CPUs). Over the
weekend I also saw only two connected users and a load of 20.  The load
is generated by smb-processes each eating up about 10%-15% CPU time
running as user root.

Watching at the root owned smb pids in top I noticed that their pids are
rapidly counting up, approxemately by a number of about 20 each second.
By trying to strace one of these processes, which mostly failed because
the process had already died when I started strace, I learned that they
only live about a second until a new one is spawned.

After searching log files I found that each of these smb process is
spawned by a windows client request that tries to authenticate via ntlm.
I interpret the log messages I found  of these smb processes that the
Windows *machine* (not the user) tries to authenticate against samba. In
the samba log files I see the messages below for many win clients:

[2017/10/02 11:07:46.987944,  2]
../source3/param/loadparm.c:2689(lp_do_section)  Processing section
"[share1]"
[2017/10/02 11:07:46.988010,  2]
../source3/param/loadparm.c:2689(lp_do_section)  Processing section
"[share2]"
....
 [2017/10/02 11:07:47.046715,  2]
../source3/auth/auth.c:315(auth_check_ntlm_password)
check_ntlm_password:  Authentication for user [HOSTNAME$] -> [HOSTNAME$]
FAILED with error NT_STATUS_NO_SUCH_USER

These log messages form a loop. This "loop" (client request, new smb
process, failed ntlm authentication, smb process dies, new client
request) repeats about once a second for each affected win client. In
summary this generates the load I see.
For each such authentication request the new smb process that is spawned
parses the whole smb.conf with all shares and then fails to do the
authentication requested by the client, that as far as I know should be
done against the windows domain controller but not against the samba
server.

All these windows clients are domain clients of our local windows domain
"MYREALM.UNI-KOBLENZ.DE" served by a real windows domain server. From
a
users point of view everything works fine allthough things could still
speed up if the load was lower.

The really strange thing about this problem is that it occured first
about 2 weeks ago, but in this time there was no new samba version
installed or any change in configuration. The time before the load went
up to 5 or 10 but not more.

Now at semester break only about 120 users are active at a time, during
semester there are usually about 300 active users. But even these 300
users did not cause a load of 50, that I can ovserve now a peek times.

Does anyone have a idea what might be going on here with these large
number of machine ntlm auth tries suddenly?

Thank you very much

Here is my smb.conf without shares:

[global]
        workgroup = MYREALM
        domain master = no
        local master = no
        preferred master = no
        ntlm auth = no
        lanman auth = no
        lm announce = no
        encrypt passwords = Yes
        unix extensions = no
        wide links = yes
        kernel oplocks = no
        oplocks = yes
        posix locking = no
        blocking locks = no
        acl allow execute always = yes
        socket options = TCP_NODELAY
        max open files = 32808
        read raw = yes
        write raw = yes
        max xmit = 262144
        dead time = 15
        getwd cache = yes
        stat cache = yes
        disable netbios = yes
        smb ports = 445

        dos charset = CP850
        unix charset = CP850
        name resolve order = host wins bcast
        passdb backend = tdbsam
        vfs objects = fileid

        realm = MYREALM.UNI-KOBLENZ.DE
        security = ADS
        map untrusted to domain = yes
        map to guest = never
        idmap config MYREALM : backend = nss
        idmap config MYREALM : range = 0-2000000
        idmap config MYREALM : read only = yes
        idmap config * : backend = tdb
        idmap config * : range = 3000000-4000000
        idmap config * : read only = no

-- 
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312
Web: http://userpages.uni-koblenz.de/~krienke
PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html

On Mon, 2 Oct 2017 14:51:54 +0200
Rainer Krienke via samba <samba at lists.samba.org> wrote:
> Hello,
> ....
>  [2017/10/02 11:07:47.046715,  2]
> ../source3/auth/auth.c:315(auth_check_ntlm_password)
> check_ntlm_password:  Authentication for user [HOSTNAME$] ->
> [HOSTNAME$] FAILED with error NT_STATUS_NO_SUCH_USER
> 
It looks fairly obvious to me, the Samba machine doesn't know the user
trying to connect.
> 
> All these windows clients are domain clients of our local windows
> domain "MYREALM.UNI-KOBLENZ.DE" served by a real windows domain
> server. From a users point of view everything works fine allthough
> things could still speed up if the load was lower.
> 
> The really strange thing about this problem is that it occured first
> about 2 weeks ago, but in this time there was no new samba version
> installed or any change in configuration. The time before the load
> went up to 5 or 10 but not more.
Has anything changed on the windows machines ? any updates etc.
> 
> Here is my smb.conf without shares:
> 
> [global]
>         workgroup = MYREALM
>         domain master = no
>         local master = no
>         preferred master = no
>         ntlm auth = no
>         lanman auth = no
>         lm announce = no
>         encrypt passwords = Yes
>         unix extensions = no
>         wide links = yes
>         kernel oplocks = no
>         oplocks = yes
>         posix locking = no
>         blocking locks = no
>         acl allow execute always = yes
>         socket options = TCP_NODELAY
>         max open files = 32808
>         read raw = yes
>         write raw = yes
>         max xmit = 262144
>         dead time = 15
>         getwd cache = yes
>         stat cache = yes
>         disable netbios = yes
>         smb ports = 445
> 
>         dos charset = CP850
>         unix charset = CP850
>         name resolve order = host wins bcast
>         passdb backend = tdbsam
>         vfs objects = fileid
> 
>         realm = MYREALM.UNI-KOBLENZ.DE
>         security = ADS
>         map untrusted to domain = yes
>         map to guest = never
>         idmap config MYREALM : backend = nss
>         idmap config MYREALM : range = 0-2000000
>         idmap config MYREALM : read only = yes
>         idmap config * : backend = tdb
>         idmap config * : range = 3000000-4000000
>         idmap config * : read only = no
> 
Is there any reason for using the idmap_nss backend ?
With this, you need users on the Samba machine with the same name as
the Domain users i.e. for DOMAIN\jsmith there must be a Unix user
called jsmith.

I would suggest you change it to:

        idmap config MYREALM : backend = rid
        idmap config MYREALM : range = 0-2000000
        idmap config * : backend = tdb
        idmap config * : range = 3000000-4000000 

This would mean the users and groups IDs would change.

I think this is what is happening, a user is trying to connect, this
user doesn't have a corresponding Unix user, so gets rejected, even
though it is a valid domain user.

Rowland

Am 02.10.2017 um 16:41 schrieb Rowland Penny via samba:
> On Mon, 2 Oct 2017 14:51:54 +0200
> Rainer Krienke via samba <samba at lists.samba.org> wrote:
> 
>> Hello,
>> ....
>>  [2017/10/02 11:07:47.046715,  2]
>> ../source3/auth/auth.c:315(auth_check_ntlm_password)
>> check_ntlm_password:  Authentication for user [HOSTNAME$] ->
>> [HOSTNAME$] FAILED with error NT_STATUS_NO_SUCH_USER
>>
> 
> It looks fairly obvious to me, the Samba machine doesn't know the user
> trying to connect.
> 
Hello,

Thanks for your answer. I doubt that this is a user authentication. On
the system with the "ntlm every second auth" problem I saw the logged
in
user had his shares connected in smbstatus, and no user would be able to
try to connect each second as the ntlm log messages indicate.

Moreover if this was a problem of a user trying to connect to a share,
then I would expect to hear complaints from exactly those users where
the connections obviously fail with the message from above. But there
are no complaints. And as far as I understand windows and samba ADS
security, authentication is done by the domaincontroller (which is not
our smb server) via kerberos and not via ntlm.

The guess of our windows admin is that the clients with this behaviour
talk to the samba server like they should do to the domain controller
for domain-client management. But on the domaincontroller there are no
error messages or hints to what this is all about.

> Is there any reason for using the idmap_nss backend ?
> With this, you need users on the Samba machine with the same name as
> the Domain users i.e. for DOMAIN\jsmith there must be a Unix user
> called jsmith.
Yes this is true. We have many people at our sites working with windows
and also people working with linux. Sometimes people are even using both
systems. So all user-ids always exist on unix and windows, so that it
does not matter on which system a file has been created/edited, it will
be available on all systems with proper ownership and permissions.
> Has anything changed on the windows machines ? any updates etc.
Regular MS patches are always installed on the windows clients. So it
might be such an patch that causes trouble, but after all you can't run
windows without them....


Any alternative theory is welcome.....

Thanks
Rainer
-- 
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312
Web: http://userpages.uni-koblenz.de/~krienke
PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html