samba - Jul 2023 - winbindd authentication fails with NT_STATUS_RPC_SEC_PKG_ERROR intermittently

On further investigation, the error that shows up in packet capture is that
the DC is returning [Fault: nca_s_fault_sec_pkg_error] for the
NetrLogonSamLogonEx call. There are no error logs (or any logs) regarding
the netlogon call failure in the netlogon logs even after enabling debug
logs in the DC. One more interesting thing is restarting the netlogon.exe
service on the DC also fixes the issue temporarily similar to restarting
the smb service

Is it possible that something is going stale in the winbindd memory/cache
that is getting fixed on these restarts? If yes, how do I go about
debugging that as it is not apparent from the logs?

P.S: We have different setups and the frequency of this error is
different in all of them. Also, there is another setup with Samba 4.7 on
Ubuntu 18.04 and everything works fine there.

On Fri, Jun 16, 2023 at 1:26?PM Bharath Bheemarasetti <
bharath.bheemarasetti at gmail.com> wrote:
> First 'winbind enum' lines, they can and do slow things down in
large
> domains and aren't required at all, getent etc will work without them.
> there are some old programs that will not work without them, but when
> was the last time you ran 'finger' for instance ?
>
> I made this change and it makes some difference but doesn't fix the
issue entirely. Earlier the auth calls used to fail in around a day which has
increased to 2 days now after which the auth calls fail with
NT_STATUS_RPC_SEC_PKG_ERROR and winbind needs to be restarted for it to work. We
use NTLMv2 for authentication and using the ntlm_auth tool
(https://www.samba.org/samba/docs/current/man-html/ntlm_auth.1.html) returns the
same NT_STATUS_RPC_SEC_PKG_ERROR error as well while wbinfo -i returns the
correct user info.
>
> Is there anything else that can be done to fix this permanently?
>
> You might also want to read the smb.conf manpage, you have lots of lines
> that I would never set.
>
> Thanks, I removed some lines which are not used anymore and will be
cleaning up others shortly.
>
>
> On Sat, Jun 3, 2023 at 1:09?PM Bharath Bheemarasetti <
> bharath.bheemarasetti at gmail.com> wrote:
>
>> A couple of things possible, from 4.8.0 winbind must be running and
your
>> smb.conf is, to be blunt, rubbish. You need to set the workgroup, you
>> need to have idmap config lines for the workgroup, the 'winbind
enum'
>> lines only slow things down and 'map untrusted to domain' has
been removed.
>>
>> Winbind is running and the workgroup was set as well. I omitted some
lines from the smb.conf shared previously as I wasn't sure if they were
relevant or not. I've added the full content below. Also share is being
accessed by a windows client which is part of the domain and it does work fine
for a few hours after restarting the smbd and winbind services. Does
'winbind enum' have any relation to that?
>>
>>
https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html#WINBINDENUMUSERS
mentions turning off 'winbind enum' can cause some problems
>>
>> *Configuration:*
>>
>> netbios name = clustF994DF
>> realm = <domain>
>>
>> bind interfaces only = yes
>> interfaces = 127.0.0.138 lo:138
>>
>> workgroup = <workgroup>
>> security = ads
>> server role = member server
>>
>> auth methods = winbind
>>
>> idmap config * : backend = tdb
>> idmap config * : range = 10000-24999999
>>
>> winbind enum users = yes
>> winbind enum groups = yes
>> usershare allow guests = no
>>
>> map untrusted to domain = Yes
>> allow trusted domains = no
>> server string = %h
>> dns proxy = no
>> log file = /var/log/samba/log.%m
>> max log size = 1000
>> panic action = /usr/share/samba/panic-action %d
>> smb ports = 1445
>> pid directory = /var/run/samba
>>
>> server min protocol = SMB2
>> strict sync = yes
>> sync always = no
>>
>> smb encrypt = auto
>>
>> aio read size = 1
>> aio write size = 1
>>
>> smb2 max read = 1048576
>> smb2 max write = 1048576
>> smb2 max trans = 1048576
>>
>> socket options = TCP_NODELAY SO_RCVBUF=10485760 SO_SNDBUF=10485760
>>
>> usershare owner only = no
>>
>> load printers = no
>> printing = bsd
>> printcap name = /dev/null
>> disable spoolss = yes
>>
>> machine password timeout = 0
>>
>> nt acl support = yes
>> vfs objects = acl_xattr
>> map acl inherit = yes
>> store dos attributes = yes
>>
>> log level = 5
>> max log size = 1000
>>
>> *Share configuration:*
>>
>>   path = <path>
>>
>>   guest ok = no
>>
>>   writeable = no
>>
>>   browseable = no
>>
>>   valid users =
"<domain>\<user>","+<domain>\<user
group>"
>>
>>   force user = root
>>
>> On Fri, Jun 2, 2023 at 3:21?AM Bharath Bheemarasetti <
>> bharath.bheemarasetti at gmail.com> wrote:
>>
>>> Hi,
>>> I recently upgraded a smb server from Ubuntu 18.04 to Ubuntu 20.04
which
>>> required the Samba version to be upgraded from 4.7.6 to 4.15.13.
>>> Post the upgrade, winbind authentication fails
>>> with  NT_STATUS_RPC_SEC_PKG_ERROR intermittently. The error goes
away on
>>> restarting the smb service but comes back after some time. There
were no
>>> isses with the setup before the upgrade.
>>> Tried clearing the cached tdb files as well but the issue still
come
>>> back after some time.
>>> <trimmed the log lines>
>>>
>>
>>> Below is the configuration:
>>> security = ads
>>> server role = member server
>>> auth methods = winbind
>>> idmap config * : backend = tdb
>>> idmap config * : range = 10000-24999999
>>> winbind enum users = yes
>>> winbind enum groups = yes
>>> usershare allow guests = no
>>> map untrusted to domain = Yes
>>> allow trusted domains = no
>>>
>>

On 02/07/2023 20:40, Bharath Bheemarasetti via samba
wrote:
> On further investigation, the error that shows up in packet capture is that
> the DC is returning [Fault: nca_s_fault_sec_pkg_error] for the
> NetrLogonSamLogonEx call. There are no error logs (or any logs) regarding
> the netlogon call failure in the netlogon logs even after enabling debug
> logs in the DC. One more interesting thing is restarting the netlogon.exe
> service on the DC also fixes the issue temporarily similar to restarting
> the smb service
> 
> Is it possible that something is going stale in the winbindd memory/cache
> that is getting fixed on these restarts? If yes, how do I go about
> debugging that as it is not apparent from the logs?
> 
> P.S: We have different setups and the frequency of this error is
> different in all of them. Also, there is another setup with Samba 4.7 on
> Ubuntu 18.04 and everything works fine there.
> 
> 
Sorry, but I think you are going to have to give us a bit more info.

What are the DC's ?
Why are you using NTLMv2 ? what is it required for ?

Did you add idmap config lines for the '<workgroup>' domain ?
Is it possible you are also using sssd or something similar ?

I think you need to describe your setup in a lot more detail.

I personally think you may be doing something in way that is been 
superseded by other methods, for instance, you normally only use 
ntlm_auth for external programs, Samba no longer requires or uses it.

Rowland

On Mon, Jul 03, 2023 at 01:10:26AM +0530, Bharath Bheemarasetti via samba
wrote:
>On further investigation, the error that shows up in packet capture is that
>the DC is returning [Fault: nca_s_fault_sec_pkg_error] for the
>NetrLogonSamLogonEx call. There are no error logs (or any logs) regarding
>the netlogon call failure in the netlogon logs even after enabling debug
>logs in the DC. One more interesting thing is restarting the netlogon.exe
>service on the DC also fixes the issue temporarily similar to restarting
>the smb service
That sounds like netlogon.exe crashing in some way. Have you
reported this to Microsoft ?

>>> What are the DC's ?
DC is a read-write windows active directory domain controller on
Windows server 2016.
>>> Why are you using NTLMv2 ? what is it required for ?
The smb client here is a windows server 2016 machine part of a domain
and the smb server is on Ubuntu 22.04. The communication between the
client and the server uses NTLMv2 while the communication between the
server and the DC happens with kerberos. The error we see is when
trying to mount the smb share on the client.
>>> Did you add idmap config lines for the '<workgroup>'
domain ?
Not yet, will add that the next time we hit the issue.
>>> Is it possible you are also using sssd or something similar ?
No, we are not using sssd.


On Mon, Jul 3, 2023 at 1:10?AM Bharath Bheemarasetti <
bharath.bheemarasetti at gmail.com> wrote:
> On further investigation, the error that shows up in packet capture is
> that the DC is returning [Fault: nca_s_fault_sec_pkg_error] for the
> NetrLogonSamLogonEx call. There are no error logs (or any logs) regarding
> the netlogon call failure in the netlogon logs even after enabling debug
> logs in the DC. One more interesting thing is restarting the netlogon.exe
> service on the DC also fixes the issue temporarily similar to restarting
> the smb service
>
> Is it possible that something is going stale in the winbindd memory/cache
> that is getting fixed on these restarts? If yes, how do I go about
> debugging that as it is not apparent from the logs?
>
> P.S: We have different setups and the frequency of this error is
> different in all of them. Also, there is another setup with Samba 4.7 on
> Ubuntu 18.04 and everything works fine there.
>
> On Fri, Jun 16, 2023 at 1:26?PM Bharath Bheemarasetti <
> bharath.bheemarasetti at gmail.com> wrote:
>
>> First 'winbind enum' lines, they can and do slow things down in
large
>> domains and aren't required at all, getent etc will work without
them.
>> there are some old programs that will not work without them, but when
>> was the last time you ran 'finger' for instance ?
>>
>> I made this change and it makes some difference but doesn't fix the
issue entirely. Earlier the auth calls used to fail in around a day which has
increased to 2 days now after which the auth calls fail with
NT_STATUS_RPC_SEC_PKG_ERROR and winbind needs to be restarted for it to work. We
use NTLMv2 for authentication and using the ntlm_auth tool
(https://www.samba.org/samba/docs/current/man-html/ntlm_auth.1.html) returns the
same NT_STATUS_RPC_SEC_PKG_ERROR error as well while wbinfo -i returns the
correct user info.
>>
>> Is there anything else that can be done to fix this permanently?
>>
>> You might also want to read the smb.conf manpage, you have lots of
lines
>> that I would never set.
>>
>> Thanks, I removed some lines which are not used anymore and will be
cleaning up others shortly.
>>
>>
>> On Sat, Jun 3, 2023 at 1:09?PM Bharath Bheemarasetti <
>> bharath.bheemarasetti at gmail.com> wrote:
>>
>>> A couple of things possible, from 4.8.0 winbind must be running and
your
>>> smb.conf is, to be blunt, rubbish. You need to set the workgroup,
you
>>> need to have idmap config lines for the workgroup, the 'winbind
enum'
>>> lines only slow things down and 'map untrusted to domain'
has been removed.
>>>
>>> Winbind is running and the workgroup was set as well. I omitted
some lines from the smb.conf shared previously as I wasn't sure if they were
relevant or not. I've added the full content below. Also share is being
accessed by a windows client which is part of the domain and it does work fine
for a few hours after restarting the smbd and winbind services. Does
'winbind enum' have any relation to that?
>>>
>>>
https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html#WINBINDENUMUSERS
mentions turning off 'winbind enum' can cause some problems
>>>
>>> *Configuration:*
>>>
>>> netbios name = clustF994DF
>>> realm = <domain>
>>>
>>> bind interfaces only = yes
>>> interfaces = 127.0.0.138 lo:138
>>>
>>> workgroup = <workgroup>
>>> security = ads
>>> server role = member server
>>>
>>> auth methods = winbind
>>>
>>> idmap config * : backend = tdb
>>> idmap config * : range = 10000-24999999
>>>
>>> winbind enum users = yes
>>> winbind enum groups = yes
>>> usershare allow guests = no
>>>
>>> map untrusted to domain = Yes
>>> allow trusted domains = no
>>> server string = %h
>>> dns proxy = no
>>> log file = /var/log/samba/log.%m
>>> max log size = 1000
>>> panic action = /usr/share/samba/panic-action %d
>>> smb ports = 1445
>>> pid directory = /var/run/samba
>>>
>>> server min protocol = SMB2
>>> strict sync = yes
>>> sync always = no
>>>
>>> smb encrypt = auto
>>>
>>> aio read size = 1
>>> aio write size = 1
>>>
>>> smb2 max read = 1048576
>>> smb2 max write = 1048576
>>> smb2 max trans = 1048576
>>>
>>> socket options = TCP_NODELAY SO_RCVBUF=10485760 SO_SNDBUF=10485760
>>>
>>> usershare owner only = no
>>>
>>> load printers = no
>>> printing = bsd
>>> printcap name = /dev/null
>>> disable spoolss = yes
>>>
>>> machine password timeout = 0
>>>
>>> nt acl support = yes
>>> vfs objects = acl_xattr
>>> map acl inherit = yes
>>> store dos attributes = yes
>>>
>>> log level = 5
>>> max log size = 1000
>>>
>>> *Share configuration:*
>>>
>>>   path = <path>
>>>
>>>   guest ok = no
>>>
>>>   writeable = no
>>>
>>>   browseable = no
>>>
>>>   valid users =
"<domain>\<user>","+<domain>\<user
group>"
>>>
>>>   force user = root
>>>
>>> On Fri, Jun 2, 2023 at 3:21?AM Bharath Bheemarasetti <
>>> bharath.bheemarasetti at gmail.com> wrote:
>>>
>>>> Hi,
>>>> I recently upgraded a smb server from Ubuntu 18.04 to Ubuntu
20.04
>>>> which required the Samba version to be upgraded from 4.7.6 to
4.15.13.
>>>> Post the upgrade, winbind authentication fails
>>>> with  NT_STATUS_RPC_SEC_PKG_ERROR intermittently. The error
goes away on
>>>> restarting the smb service but comes back after some time.
There were no
>>>> isses with the setup before the upgrade.
>>>> Tried clearing the cached tdb files as well but the issue still
come
>>>> back after some time.
>>>> <trimmed the log lines>
>>>>
>>>
>>>> Below is the configuration:
>>>> security = ads
>>>> server role = member server
>>>> auth methods = winbind
>>>> idmap config * : backend = tdb
>>>> idmap config * : range = 10000-24999999
>>>> winbind enum users = yes
>>>> winbind enum groups = yes
>>>> usershare allow guests = no
>>>> map untrusted to domain = Yes
>>>> allow trusted domains = no
>>>>
>>>