samba - Aug 2020 - PANIC running Samba as Domain Controller

We've been having issues with the samba process PANICing for some time
now (approx 24 months, though the log files have rotated so I can't
give an exact date).
For most of this time it was an annoyance - it occurred perhaps once a
fortnight and didn't cause great disruption.
Around the end of 2019 the PANIC's became more frequent. When they
occur machines attempting to authenticate seem to hang for a couple of
minutes, unaware the process they were talking to on the DC has died.
This is a particular issue for our the Windows 2012 R2 machine doing
RADIUS for our wireless controller as noone can get on the wifi until
that 2012 machine comes out of the timeout.
At their worst they have occurred every 5 minutes for about 90
minutes, though I only witnessed that one morning week commencing 3rd
August.

The PANICs occur on all DCs, but much more frequently on the one at
our hub site which has the most users, wireless controller, IIS server
authenticating webapps - presumably due to load. I'm assuming that is
why the PANICs have increased over time - increasing load.

Originally the DCs ran CentOS 6 with samba compiled from source. The
only thread I could find for this issue was
https://lists.samba.org/archive/samba/2020-January/227768.html which
appears to be the same. The advice given was that the dependencies
were likely out of date and the best option was to run packages from
apt.van-belle.nl rather than compiling from source.

I worked towards this goal slowly first replacing the hub site DC with
a CentOS 8 machine (to obtain up to date dependencies) and compiling
from source. This issue persisted. I tried initially version 4.12.5,
then rebuilt the DC with 4.9.18 (hoping the issue was introduced in
4.10) but the PANICs continued under both versions.

I then rebuilt the DC with ubuntu 20.04 using the stock 4.11.6 samba
packages, but still got the panics. Last night I added the
apt.van-belle.nl repository and upgraded to the 4.12.6 packages from
there, but this morning I have been greeted with another PANIC.

Its backtrace is below. Please advise what further troubleshooting
steps I can take to find out what is causing this issue and resolve
it.

The domain was created in 2012 with Samba 4.1.12 on CentOS 6 and
upgraded over time. There are 6 DCs, one at each site.

Many thanks,

Will

[2020/08/19 07:01:23.527871,  0]
../../source4/lib/cmdline/popt_common.c:74(popt_s4_talloc_log_fn)
  Bad talloc magic value - unknown value
[2020/08/19 07:01:23.531455,  0] ../../lib/util/fault.c:131(smb_panic_default)
  smb_panic_default: PANIC (pid 1188): Bad talloc magic value - unknown value
[2020/08/19 07:01:23.565451,  0] ../../lib/util/fault.c:264(log_stack_trace)
  BACKTRACE: 58 stack frames:
   #0 /lib/x86_64-linux-gnu/libsamba-util.so.0(log_stack_trace+0x34)
[0x7ff8ea21e624]
   #1 /lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x51) [0x7ff8ea21e741]
   #2 /lib/x86_64-linux-gnu/libtalloc.so.2(+0x3622) [0x7ff8e9f1b622]
   #3
/usr/lib/x86_64-linux-gnu/samba/libdsdb-module.so.0(dsdb_module_am_system+0x2b)
[0x7ff8e659a8bb]
   #4 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/acl.so(+0x63f5)
[0x7ff8e5ceb3f5]
   #5 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170) [0x7ff8e9eebe80]
   #6 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/descriptor.so(+0x40fb)
[0x7ff8e5cbe0fb]
   #7 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170) [0x7ff8e9eebe80]
   #8 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/extended_dn_in.so(+0x3060)
[0x7ff8e5c8f060]
   #9 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170) [0x7ff8e9eebe80]
   #10 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/asq.so(+0x170b)
[0x7ff8e61b170b]
   #11 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #12 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/server_sort.so(+0x1ae3)
[0x7ff8e59e0ae3]
   #13 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #14 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/anr.so(+0x1bfa)
[0x7ff8e5cd9bfa]
   #15 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #16
/usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/ranged_results.so(+0x1a9c)
[0x7ff8e5bf2a9c]
   #17 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #18 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/vlv.so(+0x3323)
[0x7ff8e59f1323]
   #19 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #20 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/paged_results.so(+0x314b)
[0x7ff8e5c1e14b]
   #21 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #22 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/dirsync.so(+0x2beb)
[0x7ff8e5cb2beb]
   #23 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #24 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/lazy_commit.so(+0x15cb)
[0x7ff8e5c635cb]
   #25 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #26
/usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/dsdb_notification.so(+0x1643)
[0x7ff8e5c9d643]
   #27 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #28 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/rootdse.so(+0x7ed3)
[0x7ff8e5b51ed3]
   #29 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #30 /usr/lib/x86_64-linux-gnu/ldb/modules/ldb/samba/resolve_oids.so(+0x20f3)
[0x7ff8e5b610f3]
   #31 /lib/x86_64-linux-gnu/libldb.so.2(ldb_next_request+0x170)
[0x7ff8e9eebe80]
   #32 /lib/x86_64-linux-gnu/libldb.so.2(+0x231cb) [0x7ff8e9f041cb]
   #33 /lib/x86_64-linux-gnu/libldb.so.2(ldb_request+0x1d6) [0x7ff8e9f034a6]
   #34 /usr/lib/x86_64-linux-gnu/samba/libsamdb-common.so.0(dsdb_search+0x17f)
[0x7ff8e9f6326f]
   #35 /lib/x86_64-linux-gnu/libsamdb.so.0(+0xd7b4) [0x7ff8ea1c77b4]
   #36 /lib/x86_64-linux-gnu/libsamdb.so.0(DsCrackNameOneName+0x3e3)
[0x7ff8ea1c6923]
   #37
/lib/x86_64-linux-gnu/libsamdb.so.0(dcesrv_drsuapi_CrackNamesByNameFormat+0xc0)
[0x7ff8ea1c8ef0]
   #38 /lib/x86_64-linux-gnu/libdcerpc-server.so.0(+0x28d28) [0x7ff8e65edd28]
   #39 /lib/x86_64-linux-gnu/libdcerpc-server-core.so.0(+0x82e5)
[0x7ff8e65ba2e5]
   #40 /lib/x86_64-linux-gnu/libdcerpc-binding.so.0(+0xbd6f) [0x7ff8e9756d6f]
   #41 /usr/lib/x86_64-linux-gnu/samba/libsamba-sockets.so.0(+0x8f0b)
[0x7ff8e978df0b]
   #42 /usr/lib/x86_64-linux-gnu/samba/libsamba-sockets.so.0(+0x7c8e)
[0x7ff8e978cc8e]
   #43
/lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_invoke_immediate_handler+0x14a)
[0x7ff8e9f31c4a]
   #44 /lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_immediate+0x1e)
[0x7ff8e9f31c6e]
   #45 /lib/x86_64-linux-gnu/libtevent.so.0(+0xdab0) [0x7ff8e9f37ab0]
   #46 /lib/x86_64-linux-gnu/libtevent.so.0(+0xbe3b) [0x7ff8e9f35e3b]
   #47 /lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x98)
[0x7ff8e9f30ec8]
   #48 /lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x1b)
[0x7ff8e9f3116b]
   #49 /lib/x86_64-linux-gnu/libtevent.so.0(+0xbdcb) [0x7ff8e9f35dcb]
   #50 /usr/lib/x86_64-linux-gnu/samba/process_model/prefork.so(+0x2d7b)
[0x7ff8e6688d7b]
   #51 /usr/lib/x86_64-linux-gnu/samba/process_model/prefork.so(+0x322a)
[0x7ff8e668922a]
   #52 /usr/lib/x86_64-linux-gnu/samba/process_model/prefork.so(+0x3484)
[0x7ff8e6689484]
   #53 /usr/lib/x86_64-linux-gnu/samba/libservice.so.0(task_server_startup+0x60)
[0x7ff8ea2bc2f0]
   #54
/usr/lib/x86_64-linux-gnu/samba/libservice.so.0(server_service_startup+0x9a)
[0x7ff8ea2bab9a]
   #55 samba: task[rpc] pre-forked worker(0)(+0x5f05) [0x559a98fdef05]
   #56 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7ff8e9cd90b3]
   #57 samba: task[rpc] pre-forked worker(0)(_start+0x2e) [0x559a98fdde2e]

On 19/08/2020 08:26, William Ross via samba wrote:
> We've been having issues with the samba process PANICing for some time
> now (approx 24 months, though the log files have rotated so I can't
> give an exact date).
>
First, Samba should not panic, but I noticed the mention of a Windows 
2012R2 machine, how is this joined to the domain ? as a DC or a domain 
member ?

If it is a DC, this will probably mean that the schema was upgraded to 
version 69 and this is still regarded as experimental, though this may 
have nothing to do with your problem.

I think you need to do a bit of work so that Samba can help you, get 
level 10 logs and traces (note you need to add the debug packages)

Rowland

On Wed, 19 Aug 2020 at 09:53, Rowland penny via samba
<samba at lists.samba.org> wrote:
> First, Samba should not panic, but I noticed the mention of a Windows
> 2012R2 machine, how is this joined to the domain ? as a DC or a domain
> member ?
The domain controllers are all Samba domain controllers, the 2012 R2
machines are domain members only.
> I think you need to do a bit of work so that Samba can help you, get
> level 10 logs and traces (note you need to add the debug packages)
>
> Rowland
Thanks for the prompt reply. When you say debug packages which do you
mean? I can't see any with 'debug' in the name on the
apt.van-belle.nl
repo.

I've enabled level 10 debug logging as per
https://wiki.samba.org/index.php/Client_specific_logging (though I
didn't configure it per client) and generated a log file of 1GB in 15
minutes. The local shared drive (hosted on a QNAP domain member)
started to become unable to authenticate users (I think due to system
load on the domain controller). I may need to run this for a couple of
days to catch the next PANIC (they are intermittent). Is there any way
I could tweak the config to reduce the load on the domain controller
whilst still getting useful debug information? The precise logging
lines in my smb.conf are:

  log level = 10
  max log size = 0
  logging = syslog at 10 file at 1
  log file = /dev/null
  # add the pid to the log
  debug pid = yes
  # add the uid to the log
  debug uid = yes
  # add the debug class to the log
  debug class = yes
  # add microsecond resolution to timestamp
  debug hires timestamp = yes

(Note that the 'logging' line doesn't appear to work. Regardless of
the number after the @ symbol it logs at the level specified by 'log
level'. The machine doesn't have enough disk space to handle the logs,
so I've set the log file to /dev/null and configured a dedicated
machine to receive them. I was hoping to be able to keep the file
logging at level 1 to provide an easy way to see when the PANICs
happen (given how huge the log file becomes at level 10)).

Will