samba - May 2023 - On Debian 12: nsupdate (as called from samba_dnsupdate) crashes named/bind9_dlz

On 20/05/2023 04:44, Steven Monai via samba wrote:
> Thanks. With this new info,
It isn't new, it is in the wiki.
> I re-ran my test setup from the beginning: 
> destroyed and reprovisioned the VMs dc33 and dc34 running Debian 12; 
> provisioned a new AD domain on dc33 with 'samba-tool domain provision 
> DC'; and then joined dc34 as a DC with 'samba-tool domain join
DC'.
> 
> Once again, the new domain on dc33 seems to be correct and functional. 
> However, once again, the necessary DNS records are not created for dc34 
> when it joins the domain.
When you provision a domain, most, if not all, of the required dns 
records are created. When you join another DC, only a few are created, 
the rest are created by samba_dnsupdate when Samba first starts, or that 
is how it is supposed to work.
> It seems samba_dnsudpate still does not work, 
> even with the updated name resolver config.
Problem is, it works for myself, I am still on bullseye, using Samba 
from backports, so have the same Samba version as you 4.17.8 . What is 
different is the version of nsupdate, mine comes from bind9-dnsutils 
9.16.37, yours is probably from 9.18.12
> 
> Here is an abbreviated snippet of the output from the command line on 
> dc34, after the domain join:
> ------------------------------------------------------------------------
> dc34:~# samba_dnsupdate --verbose
> IPs: ['10.150.10.34']
> ...
> 22 DNS updates and 0 DNS deletes needed
> Successfully obtained Kerberos ticket to DNS/dc34.ttwo.ad.example.org as 
> DC34$
> update(nsupdate): NS ttwo.ad.example.org dc34.ttwo.ad.example.org
> Calling nsupdate for NS ttwo.ad.example.org dc34.ttwo.ad.example.org (add)
> Successfully obtained Kerberos ticket to DNS/dc34.ttwo.ad.example.org as 
> DC34$
> Outgoing update query:
> ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:????? 0
> ;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
> ;; UPDATE SECTION:
> ttwo.ad.example.org.???? 900???? IN????? NS????? dc34.ttwo.ad.example.org.
> 
> ; Communication with 10.150.10.34#53 failed: end of file
> Failed nsupdate: 2
> ...
> (...similar failure of all successive zone update attempts...)
> ...
> Failed update of 22 entries
> ------------------------------------------------------------------------
> 
> And here is a snippet of the resulting log from the named server that is 
> contacted (this time on dc34, not dc33):
> ------------------------------------------------------------------------
> dc34:~# journalctl -u named.service
> ...
> May 19 10:18:30 dc34 named[4308]: samba_dlz: allowing update of 
> signer=DC34\$\@TTWO.AD.example.org name=ttwo.ad.example.org 
> tcpaddr=10.150.10.34 type=NS 
> key=1542098645.sig-dc34.ttwo.ad.example.org/159/0
> May 19 10:18:30 dc34 named[4308]: samba_dlz: starting transaction on 
> zone ttwo.ad.example.org
> May 19 10:18:30 dc34 named[4308]: client @0x7f272bffe368 
> 10.150.10.34#39821/key DC34\$\@TTWO.AD.example.org: updating zone 
> 'ttwo.ad.example.org/NONE': adding an RR at
'ttwo.ad.example.org' NS
> dc34.ttwo.ad.example.org.
> May 19 10:18:30 dc34 named[4308]: name.c:664: REQUIRE(((name1) != ((void 
> *)0) && ((const isc__magic_t *)(name1))->magic ==
((('D') << 24 | ('N')
> << 16 | ('S') << 8 | ('n'))))) failed, back
trace
> May 19 10:18:30 dc34 named[4308]: /usr/sbin/named(+0x235e4) 
> [0x556e2d6cf5e4]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(isc_assertion_failed+0xa)
[0x7f2735239a5a]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libdns-9.18.12-1-Debian.so(dns_name_equal+0x179) 
> [0x7f2734e999d9]
> May 19 10:18:30 dc34 named[4308]: 
>
/usr/lib/x86_64-linux-gnu/samba/bind9/dlz_bind9_18.so(dlz_addrdataset+0x1c4)
[0x7f2733a8cb54]
> May 19 10:18:30 dc34 named[4308]: /usr/sbin/named(+0x212e4) 
> [0x556e2d6cd2e4]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libdns-9.18.12-1-Debian.so(+0x12e4c4) 
> [0x7f2734f2e4c4]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libdns-9.18.12-1-Debian.so(+0x4ec17) [0x7f2734e4ec17]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libns-9.18.12-1-Debian.so(+0x31dca) [0x7f27357f6dca]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libns-9.18.12-1-Debian.so(+0x35466) [0x7f27357fa466]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(isc_task_run+0x113) 
> [0x7f2735258a43]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(+0x26cb2) [0x7f2735226cb2]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(+0x27337) [0x7f2735227337]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(+0x27e73) [0x7f2735227e73]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libuv.so.1(+0xf09d) [0x7f273516d09d]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libuv.so.1(+0x22e3c) [0x7f2735180e3c]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libuv.so.1(uv_run+0xc4) [0x7f273516d9e4]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(+0x27654) [0x7f2735227654]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(isc__trampoline_run+0x15)
[0x7f2735261575]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libc.so.6(+0x88fd4) [0x7f27344fbfd4]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libc.so.6(+0x1095bc) [0x7f273457c5bc]
> May 19 10:18:30 dc34 named[4308]: exiting (due to assertion failure)
> May 19 10:18:30 dc34 systemd[1]: named.service: Main process exited, 
> code=dumped, status=6/ABRT
> May 19 10:18:30 dc34 systemd[1]: named.service: Failed with result 
> 'core-dump'.
> May 19 10:18:30 dc34 systemd[1]: named.service: Scheduled restart job, 
> restart counter is at 1.
> May 19 10:18:30 dc34 systemd[1]: Stopped named.service - BIND Domain 
> Name Server.
> May 19 10:18:30 dc34 systemd[1]: Starting named.service - BIND Domain 
> Name Server...
> May 19 10:18:30 dc34 named[4319]: starting BIND 9.18.12-1-Debian 
> (Extended Support Version) <id:>
> May 19 10:18:30 dc34 named[4319]: running on Linux x86_64 6.1.0-9-amd64 
> #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08)
> ...
> (...repeat assertion-failure/core-dump/daemon-restart for every nsupdate 
> attempt...)
> ...
> ------------------------------------------------------------------------
> 
> The immediate cause of the crashes is clearly the assertion-failure 
> reported in the log.
Yes, it also looks like it is named that is crashing, not Samba.
> 
> I found an open bug in bugzilla that reports a very similar assertion 
> failure: "Bug 14030 - named crashes on DLZ zone update" 
> (https://bugzilla.samba.org/show_bug.cgi?id=14030). Any chance this Bug 
> is related to what I'm seeing?
That appears to be a Samba problem, whilst yours appears to possibly be 
a Bind9 problem.

Are you running Bind9 as the dns server ?
If so, please post the following files (inline, do not attach them, this 
list strips attachments):

/etc/bind/named.conf
/etc/bind/named.conf.options
/etc/bind/named.conf.local
/etc/bind/named.conf.default-zones
/etc/samba/smb.conf

Rowland

On 2023-05-20 1:43 a.m., Rowland Penny via samba wrote:
> On 20/05/2023 04:44, Steven Monai via samba wrote:
>> I found an open bug in bugzilla that reports a very similar assertion 
>> failure: "Bug 14030 - named crashes on DLZ zone update" 
>> (https://bugzilla.samba.org/show_bug.cgi?id=14030). Any chance this 
>> Bug is related to what I'm seeing?
> 
> That appears to be a Samba problem, whilst yours appears to possibly be 
> a Bind9 problem.
Maybe. But nevermind that Bug; it appears to be FreeBSD-specific, and I 
am using Linux (more specifically Debian, on amd64 architecture).

To clarify my context: I have an AD domain setup---currently working on 
Debian Bullseye (and Buster)---which consists of two Samba DCs that use 
the BIND9_DLZ backend for DNS. This setup is deployed in production and 
is functioning well at numerous sites in my organization right now.

Unfortunately, my "known good" Bullseye/Buster AD setup fails to work 
when applied to Bookworm servers. That specific failure is what I've 
been trying to describe in this thread so far (possibly not adequately, 
but I'm doing my best to answer all questions).

Ultimately, I would like to arrive at a fully-functioning Bookworm 
setup, since it is my organization's policy to run its systems on Debian 
Stable, and Bookworm is due to become Stable very soon (on June 10th, 
about 3 weeks from now).

Anyway, the requested configuration files are pasted inline below.
> /etc/bind/named.conf
------------------------------------------------------------------------
// This is the primary configuration file for the BIND DNS server named.
//
// Please read /usr/share/doc/bind9/README.Debian for information on the
// structure of BIND configuration files in Debian, *BEFORE* you customize
// this configuration file.
//
// If you are just adding zones, please do that in 
/etc/bind/named.conf.local

include "/etc/bind/named.conf.options";
include "/etc/bind/named.conf.local";
include "/etc/bind/named.conf.default-zones";
include "/var/lib/samba/bind-dns/named.conf";
------------------------------------------------------------------------
> /etc/bind/named.conf.options
------------------------------------------------------------------------
options {
	directory "/var/cache/bind";

	// If there is a firewall between you and nameservers you want
	// to talk to, you may need to fix the firewall to allow multiple
	// ports to talk.  See http://www.kb.cert.org/vuls/id/800113

	// If your ISP provided one or more IP addresses for stable
	// nameservers, you probably want to use them as forwarders.
	// Uncomment the following block, and insert the addresses replacing
	// the all-0's placeholder.

	// forwarders {
	// 	0.0.0.0;
	// };

	//=======================================================================	// If
BIND logs error messages about the root key being expired,
	// you will need to update your keys.  See https://www.isc.org/bind-keys
	//=======================================================================
dnssec-validation auto;

	listen-on-v6 { any; };
	tkey-gssapi-keytab "/var/lib/samba/bind-dns/dns.keytab";
	minimal-responses yes;
	allow-transfer { localhost; };
	allow-query { localhost; localnets; !10.150.40.0/22; 10.150.0.0/16; };
};
------------------------------------------------------------------------
> /etc/bind/named.conf.local
------------------------------------------------------------------------
//
// Do any local configuration here
//

// Consider adding the 1918 zones here, if they are not used in your
// organization
//include "/etc/bind/zones.rfc1918";

------------------------------------------------------------------------
> /etc/bind/named.conf.default-zones
------------------------------------------------------------------------
// prime the server with knowledge of the root servers
zone "." {
	type hint;
	file "/usr/share/dns/root.hints";
};

// be authoritative for the localhost forward and reverse zones, and for
// broadcast zones as per RFC 1912

zone "localhost" {
	type master;
	file "/etc/bind/db.local";
};

zone "127.in-addr.arpa" {
	type master;
	file "/etc/bind/db.127";
};

zone "0.in-addr.arpa" {
	type master;
	file "/etc/bind/db.0";
};

zone "255.in-addr.arpa" {
	type master;
	file "/etc/bind/db.255";
};


------------------------------------------------------------------------
> /etc/samba/smb.conf
------------------------------------------------------------------------
# Global parameters
[global]
	bind interfaces only = Yes
	disable netbios = Yes
	dns zone transfer clients allow = 127.0.0.0/8 ::1/128
	interfaces = lo enp1s0
	log level = 1 auth_json_audit:5
	netbios name = DC34
	ntlm auth = mschapv2-and-ntlmv2-only
	realm = TTWO.AD.EXAMPLE.ORG
	server role = active directory domain controller
	server services = s3fs, rpc, nbt, wrepl, ldap, cldap, kdc, drepl, 
winbindd, ntp_signd, kcc, dnsupdate
	winbind separator = /
	workgroup = TTWO
	idmap_ldb:use rfc2307  = yes

[sysvol]
	path = /var/lib/samba/sysvol
	read only = No

[netlogon]
	path = /var/lib/samba/sysvol/ttwo.ad.example.org/scripts
	read only = No
------------------------------------------------------------------------

--
Thanks,
-S.M.