samba - May 2023 - On Debian 12: nsupdate (as called from samba_dnsupdate) crashes named/bind9_dlz

On 2023-05-18 11:24 p.m., Rowland Penny via samba wrote:
> On 19/05/2023 03:57, Steven Monai via samba wrote:
>> To be concrete: What do you recommend should be the contents of the 
>> respective /etc/resolv.conf files in my test?
> In resolv.conf, 'domain' and 'search' are mutually
exclusive and the
> last one wins, as you need 'search', I would remove the
'domain' line.
> 
> As for the nameservers, I would switch them around on each DC, so that 
> the DC used itself for the nameserver.
> 
> This means:
> 
> On DC33
> 
> search ttwo.ad.example.org
> nameserver 10.150.10.33
> nameserver 10.150.10.34
> 
> ON DC34
> 
> search ttwo.ad.examole.org
> nameserver 10.150.10.34
> nameserver 10.150.10.33
Thanks. With this new info, I re-ran my test setup from the beginning: 
destroyed and reprovisioned the VMs dc33 and dc34 running Debian 12; 
provisioned a new AD domain on dc33 with 'samba-tool domain provision 
DC'; and then joined dc34 as a DC with 'samba-tool domain join DC'.

Once again, the new domain on dc33 seems to be correct and functional. 
However, once again, the necessary DNS records are not created for dc34 
when it joins the domain. It seems samba_dnsudpate still does not work, 
even with the updated name resolver config.

Here is an abbreviated snippet of the output from the command line on 
dc34, after the domain join:
------------------------------------------------------------------------
dc34:~# samba_dnsupdate --verbose
IPs: ['10.150.10.34']
...
22 DNS updates and 0 DNS deletes needed
Successfully obtained Kerberos ticket to DNS/dc34.ttwo.ad.example.org as 
DC34$
update(nsupdate): NS ttwo.ad.example.org dc34.ttwo.ad.example.org
Calling nsupdate for NS ttwo.ad.example.org dc34.ttwo.ad.example.org (add)
Successfully obtained Kerberos ticket to DNS/dc34.ttwo.ad.example.org as 
DC34$
Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; UPDATE SECTION:
ttwo.ad.example.org.     900     IN      NS      dc34.ttwo.ad.example.org.

; Communication with 10.150.10.34#53 failed: end of file
Failed nsupdate: 2
...
(...similar failure of all successive zone update attempts...)
...
Failed update of 22 entries
------------------------------------------------------------------------

And here is a snippet of the resulting log from the named server that is 
contacted (this time on dc34, not dc33):
------------------------------------------------------------------------
dc34:~# journalctl -u named.service
...
May 19 10:18:30 dc34 named[4308]: samba_dlz: allowing update of 
signer=DC34\$\@TTWO.AD.example.org name=ttwo.ad.example.org 
tcpaddr=10.150.10.34 type=NS 
key=1542098645.sig-dc34.ttwo.ad.example.org/159/0
May 19 10:18:30 dc34 named[4308]: samba_dlz: starting transaction on 
zone ttwo.ad.example.org
May 19 10:18:30 dc34 named[4308]: client @0x7f272bffe368 
10.150.10.34#39821/key DC34\$\@TTWO.AD.example.org: updating zone 
'ttwo.ad.example.org/NONE': adding an RR at
'ttwo.ad.example.org' NS
dc34.ttwo.ad.example.org.
May 19 10:18:30 dc34 named[4308]: name.c:664: REQUIRE(((name1) != ((void 
*)0) && ((const isc__magic_t *)(name1))->magic == ((('D')
<< 24 | ('N')
<< 16 | ('S') << 8 | ('n'))))) failed, back trace
May 19 10:18:30 dc34 named[4308]: /usr/sbin/named(+0x235e4) [0x556e2d6cf5e4]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(isc_assertion_failed+0xa) 
[0x7f2735239a5a]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libdns-9.18.12-1-Debian.so(dns_name_equal+0x179) 
[0x7f2734e999d9]
May 19 10:18:30 dc34 named[4308]: 
/usr/lib/x86_64-linux-gnu/samba/bind9/dlz_bind9_18.so(dlz_addrdataset+0x1c4) 
[0x7f2733a8cb54]
May 19 10:18:30 dc34 named[4308]: /usr/sbin/named(+0x212e4) [0x556e2d6cd2e4]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libdns-9.18.12-1-Debian.so(+0x12e4c4) [0x7f2734f2e4c4]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libdns-9.18.12-1-Debian.so(+0x4ec17) [0x7f2734e4ec17]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libns-9.18.12-1-Debian.so(+0x31dca) [0x7f27357f6dca]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libns-9.18.12-1-Debian.so(+0x35466) [0x7f27357fa466]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(isc_task_run+0x113) 
[0x7f2735258a43]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(+0x26cb2) [0x7f2735226cb2]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(+0x27337) [0x7f2735227337]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(+0x27e73) [0x7f2735227e73]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libuv.so.1(+0xf09d) [0x7f273516d09d]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libuv.so.1(+0x22e3c) [0x7f2735180e3c]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libuv.so.1(uv_run+0xc4) [0x7f273516d9e4]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(+0x27654) [0x7f2735227654]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(isc__trampoline_run+0x15) 
[0x7f2735261575]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libc.so.6(+0x88fd4) [0x7f27344fbfd4]
May 19 10:18:30 dc34 named[4308]: 
/lib/x86_64-linux-gnu/libc.so.6(+0x1095bc) [0x7f273457c5bc]
May 19 10:18:30 dc34 named[4308]: exiting (due to assertion failure)
May 19 10:18:30 dc34 systemd[1]: named.service: Main process exited, 
code=dumped, status=6/ABRT
May 19 10:18:30 dc34 systemd[1]: named.service: Failed with result 
'core-dump'.
May 19 10:18:30 dc34 systemd[1]: named.service: Scheduled restart job, 
restart counter is at 1.
May 19 10:18:30 dc34 systemd[1]: Stopped named.service - BIND Domain 
Name Server.
May 19 10:18:30 dc34 systemd[1]: Starting named.service - BIND Domain 
Name Server...
May 19 10:18:30 dc34 named[4319]: starting BIND 9.18.12-1-Debian 
(Extended Support Version) <id:>
May 19 10:18:30 dc34 named[4319]: running on Linux x86_64 6.1.0-9-amd64 
#1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08)
...
(...repeat assertion-failure/core-dump/daemon-restart for every nsupdate 
attempt...)
...
------------------------------------------------------------------------

The immediate cause of the crashes is clearly the assertion-failure 
reported in the log.

I found an open bug in bugzilla that reports a very similar assertion 
failure: "Bug 14030 - named crashes on DLZ zone update" 
(https://bugzilla.samba.org/show_bug.cgi?id=14030). Any chance this Bug 
is related to what I'm seeing?

--
-S.M.

On 20/05/2023 04:44, Steven Monai via samba wrote:
> Thanks. With this new info,
It isn't new, it is in the wiki.
> I re-ran my test setup from the beginning: 
> destroyed and reprovisioned the VMs dc33 and dc34 running Debian 12; 
> provisioned a new AD domain on dc33 with 'samba-tool domain provision 
> DC'; and then joined dc34 as a DC with 'samba-tool domain join
DC'.
> 
> Once again, the new domain on dc33 seems to be correct and functional. 
> However, once again, the necessary DNS records are not created for dc34 
> when it joins the domain.
When you provision a domain, most, if not all, of the required dns 
records are created. When you join another DC, only a few are created, 
the rest are created by samba_dnsupdate when Samba first starts, or that 
is how it is supposed to work.
> It seems samba_dnsudpate still does not work, 
> even with the updated name resolver config.
Problem is, it works for myself, I am still on bullseye, using Samba 
from backports, so have the same Samba version as you 4.17.8 . What is 
different is the version of nsupdate, mine comes from bind9-dnsutils 
9.16.37, yours is probably from 9.18.12
> 
> Here is an abbreviated snippet of the output from the command line on 
> dc34, after the domain join:
> ------------------------------------------------------------------------
> dc34:~# samba_dnsupdate --verbose
> IPs: ['10.150.10.34']
> ...
> 22 DNS updates and 0 DNS deletes needed
> Successfully obtained Kerberos ticket to DNS/dc34.ttwo.ad.example.org as 
> DC34$
> update(nsupdate): NS ttwo.ad.example.org dc34.ttwo.ad.example.org
> Calling nsupdate for NS ttwo.ad.example.org dc34.ttwo.ad.example.org (add)
> Successfully obtained Kerberos ticket to DNS/dc34.ttwo.ad.example.org as 
> DC34$
> Outgoing update query:
> ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:????? 0
> ;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
> ;; UPDATE SECTION:
> ttwo.ad.example.org.???? 900???? IN????? NS????? dc34.ttwo.ad.example.org.
> 
> ; Communication with 10.150.10.34#53 failed: end of file
> Failed nsupdate: 2
> ...
> (...similar failure of all successive zone update attempts...)
> ...
> Failed update of 22 entries
> ------------------------------------------------------------------------
> 
> And here is a snippet of the resulting log from the named server that is 
> contacted (this time on dc34, not dc33):
> ------------------------------------------------------------------------
> dc34:~# journalctl -u named.service
> ...
> May 19 10:18:30 dc34 named[4308]: samba_dlz: allowing update of 
> signer=DC34\$\@TTWO.AD.example.org name=ttwo.ad.example.org 
> tcpaddr=10.150.10.34 type=NS 
> key=1542098645.sig-dc34.ttwo.ad.example.org/159/0
> May 19 10:18:30 dc34 named[4308]: samba_dlz: starting transaction on 
> zone ttwo.ad.example.org
> May 19 10:18:30 dc34 named[4308]: client @0x7f272bffe368 
> 10.150.10.34#39821/key DC34\$\@TTWO.AD.example.org: updating zone 
> 'ttwo.ad.example.org/NONE': adding an RR at
'ttwo.ad.example.org' NS
> dc34.ttwo.ad.example.org.
> May 19 10:18:30 dc34 named[4308]: name.c:664: REQUIRE(((name1) != ((void 
> *)0) && ((const isc__magic_t *)(name1))->magic ==
((('D') << 24 | ('N')
> << 16 | ('S') << 8 | ('n'))))) failed, back
trace
> May 19 10:18:30 dc34 named[4308]: /usr/sbin/named(+0x235e4) 
> [0x556e2d6cf5e4]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(isc_assertion_failed+0xa)
[0x7f2735239a5a]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libdns-9.18.12-1-Debian.so(dns_name_equal+0x179) 
> [0x7f2734e999d9]
> May 19 10:18:30 dc34 named[4308]: 
>
/usr/lib/x86_64-linux-gnu/samba/bind9/dlz_bind9_18.so(dlz_addrdataset+0x1c4)
[0x7f2733a8cb54]
> May 19 10:18:30 dc34 named[4308]: /usr/sbin/named(+0x212e4) 
> [0x556e2d6cd2e4]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libdns-9.18.12-1-Debian.so(+0x12e4c4) 
> [0x7f2734f2e4c4]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libdns-9.18.12-1-Debian.so(+0x4ec17) [0x7f2734e4ec17]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libns-9.18.12-1-Debian.so(+0x31dca) [0x7f27357f6dca]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libns-9.18.12-1-Debian.so(+0x35466) [0x7f27357fa466]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(isc_task_run+0x113) 
> [0x7f2735258a43]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(+0x26cb2) [0x7f2735226cb2]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(+0x27337) [0x7f2735227337]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(+0x27e73) [0x7f2735227e73]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libuv.so.1(+0xf09d) [0x7f273516d09d]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libuv.so.1(+0x22e3c) [0x7f2735180e3c]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libuv.so.1(uv_run+0xc4) [0x7f273516d9e4]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(+0x27654) [0x7f2735227654]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libisc-9.18.12-1-Debian.so(isc__trampoline_run+0x15)
[0x7f2735261575]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libc.so.6(+0x88fd4) [0x7f27344fbfd4]
> May 19 10:18:30 dc34 named[4308]: 
> /lib/x86_64-linux-gnu/libc.so.6(+0x1095bc) [0x7f273457c5bc]
> May 19 10:18:30 dc34 named[4308]: exiting (due to assertion failure)
> May 19 10:18:30 dc34 systemd[1]: named.service: Main process exited, 
> code=dumped, status=6/ABRT
> May 19 10:18:30 dc34 systemd[1]: named.service: Failed with result 
> 'core-dump'.
> May 19 10:18:30 dc34 systemd[1]: named.service: Scheduled restart job, 
> restart counter is at 1.
> May 19 10:18:30 dc34 systemd[1]: Stopped named.service - BIND Domain 
> Name Server.
> May 19 10:18:30 dc34 systemd[1]: Starting named.service - BIND Domain 
> Name Server...
> May 19 10:18:30 dc34 named[4319]: starting BIND 9.18.12-1-Debian 
> (Extended Support Version) <id:>
> May 19 10:18:30 dc34 named[4319]: running on Linux x86_64 6.1.0-9-amd64 
> #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 (2023-05-08)
> ...
> (...repeat assertion-failure/core-dump/daemon-restart for every nsupdate 
> attempt...)
> ...
> ------------------------------------------------------------------------
> 
> The immediate cause of the crashes is clearly the assertion-failure 
> reported in the log.
Yes, it also looks like it is named that is crashing, not Samba.
> 
> I found an open bug in bugzilla that reports a very similar assertion 
> failure: "Bug 14030 - named crashes on DLZ zone update" 
> (https://bugzilla.samba.org/show_bug.cgi?id=14030). Any chance this Bug 
> is related to what I'm seeing?
That appears to be a Samba problem, whilst yours appears to possibly be 
a Bind9 problem.

Are you running Bind9 as the dns server ?
If so, please post the following files (inline, do not attach them, this 
list strips attachments):

/etc/bind/named.conf
/etc/bind/named.conf.options
/etc/bind/named.conf.local
/etc/bind/named.conf.default-zones
/etc/samba/smb.conf

Rowland