samba - Jun 2018 - BIND9_DLZ: TKEY is unacceptable - depending on the name server

Dear Samba experts, 

Since a couple of days I am trying to fix my domain. 
I have each two ADDCs on raspis on two sites. One is running on Raspian and
works fine. The other three are on Gentoo and something is broken there.
When I point the name resolution in resolv.conf to the Raspian machine the
dynamic updates are just working fine:


# horus /srv/samba/demoshare # samba_dnsupdate --verbose --all-names
# IPs: ['192.168.41.25']
# force update: A horus.samdom.com 192.168.41.25
# force update: NS samdom.com horus.samdom.com
# force update: NS _msdcs.samdom.com horus.samdom.com
# force update: A samdom.com 192.168.41.25
# .....
# 29 DNS updates and 0 DNS deletes needed
# Successfully obtained Kerberos ticket to DNS/charon.samdom.com as HORUS$
# update(nsupdate): A horus.samdom.com 192.168.41.25
# Calling nsupdate for A horus.samdom.com 192.168.41.25 (add)
# Successfully obtained Kerberos ticket to DNS/charon.samdom.com as HORUS$
# Outgoing update query:
# ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
# ;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
# ;; UPDATE SECTION:
# horus.samdom.com.    900     IN      A       192.168.41.25
#
# update(nsupdate): NS samdom.com horus.samdom.com
# .....


Now I edit resolv.conf to point to the ADDC charon at 192.168.11.205, 
and the Kerberos ticket is now obtained by DNS/horus.samdom.com, which 
is actually on of the Gentoo machines, and even though it states the 
Ticket was granted successfully, the update fails.


# horus ~ # samba_dnsupdate --verbose --all-names
# IPs: ['192.168.41.25']
# force update: A horus.samdom.com 192.168.41.25
# .....
# 29 DNS updates and 0 DNS deletes needed
# Successfully obtained Kerberos ticket to DNS/horus.samdom.com as HORUS$
# update(nsupdate): A horus.samdom.com 192.168.41.25
# Calling nsupdate for A horus.samdom.com 192.168.41.25 (add)
# Successfully obtained Kerberos ticket to DNS/horus.samdom.com as HORUS$
# Outgoing update query:
# ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
# ;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
# ;; UPDATE SECTION:
# horus.samdom.com.    900     IN      A       192.168.41.25
# 
# dns_tkey_gssnegotiate: TKEY is unacceptable
# Failed nsupdate: 1
# update(nsupdate): NS samdom.com horus.samdom.com
# .....


Needless to say, that tried to generate new keytabs. I demoted machines 
and re-joined them, but the issue persists. Actually there is samba-4.8.3 
on all machines, and the ldb/tdb/tevent/talloc in the same version as 
bundled with samba-4.8.3. Raspbian has a pretty old Bind  9.10.3-P4. 
On Gentoo I tried 9.11.3 and 9.11.2_p1. 

What I need first is a tip for an efficient setting for debugging it. 
Is there a way to have a look on the granted tickets? There must be 
some difference. 

I examined the output from named, but I could not see something fishy 
there. 

This one works OK: 
> root at charon:/usr/local/samba/private# named -V
> BIND 9.10.3-P4-Raspbian <id:ebd72b3>
> built by make with '--prefix=/usr'
'--mandir=/usr/share/man'
> '--libdir=/usr/lib/arm-linux-gnueabihf'
'--infodir=/usr/share/info'
> '--sysconfdir=/etc/bind' '--with-python=python3'
'--localstatedir=/'
> '--enable-threads' '--enable-largefile'
'--with-libtool'
> '--enable-shared' '--enable-static'
'--with-gost=no'
> '--with-openssl=/usr' '--with-gssapi=/usr'
'--with-gnu-ld'
> '--with-geoip=/usr' '--with-atf=no' '--enable-ipv6'
'--enable-rrl'
> '--enable-filter-aaaa' '--enable-native-pkcs11' 
> '--with-pkcs11=/usr/lib/arm-linux-gnueabihf/softhsm/libsofthsm2.so'
> '--with-randomdev=/dev/urandom' 'CFLAGS=-g -O2 
> -fdebug-prefix-map=/build/bind9-6GG44j/bind9-9.10.3.dfsg.P4=. 
> -fstack-protector-strong -Wformat -Werror=format-security 
> -fno-strict-aliasing -fno-delete-null-pointer-checks 
> -DNO_VERSION_DATE -DDIG_SIGCHASE' 'LDFLAGS=-Wl,-z,relro
-Wl,-z,now'
> 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'

And this is not working (on the Gentoo machine):
> horus /etc/portage # named -V
> BIND 9.11.3 (Extended Support Version) <id:a375815>
> running on Linux armv7l 4.4.136-695e41116993e0a4f080354e72f13d91-0 #1 
> SMP Thu Jun 14 14:09:46 CEST 2018
> built by make with '--prefix=/usr'
'--build=armv7a-hardfloat-linux-gnueabi'
> '--host=armv7a-hardfloat-linux-gnueabi'
'--mandir=/usr/share/man'
> '--infodir=/usr/share/info' '--datadir=/usr/share'
'--sysconfdir=/etc'
> '--localstatedir=/var/lib' '--libdir=/usr/lib'
'--sysconfdir=/etc/bind'
> '--localstatedir=/var' '--with-libtool'
'--enable-full-report'
> '--without-readline' '--enable-linux-caps'
'--disable-filter-aaaa'
> '--disable-fixed-rrset' '--disable-ipv6'
'--disable-rpz-nsdname'
> '--disable-rpz-nsip' '--disable-seccomp'
'--enable-threads'
> '--with-dlz-bdb' '--with-dlopen'
'--with-dlz-filesystem' '--with-dlz-stub'
> '--with-gost' '--with-gssapi' '--without-idn'
'--without-libjson'
> '--without-dlz-ldap' '--without-dlz-mysql'
'--without-dlz-odbc'
> '--without-dlz-postgres' '--without-lmdb'
'--with-python' '--with-ecdsa'
> '--with-openssl=/usr' '--without-libxml2'
'--with-zlib'
> '--with-randomdev=/dev/random'
'build_alias=armv7a-hardfloat-linux-gnueabi'
> 'host_alias=armv7a-hardfloat-linux-gnueabi' 'CFLAGS=-O2 -pipe 
> -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard -I/usr/include/db5.3' 
> 'LDFLAGS=-Wl,-O1 -Wl,--as-needed'
> compiled by GCC 6.4.0
> compiled with OpenSSL version: OpenSSL 1.0.2o  27 Mar 2018
> linked to OpenSSL version: OpenSSL 1.0.2o  27 Mar 2018
> compiled with zlib version: 1.2.11
> linked to zlib version: 1.2.11
> threads support is enabled

Thank You in advance and best regards
Peter

On Sat, 30 Jun 2018 16:01:10 +0200 (CEST)
Peter Serbe via samba <samba at lists.samba.org> wrote:
> Dear Samba experts, 
> 
> Since a couple of days I am trying to fix my domain. 
> I have each two ADDCs on raspis on two sites. One is running on
> Raspian and works fine. The other three are on Gentoo and something
> is broken there. When I point the name resolution in resolv.conf to
> the Raspian machine the dynamic updates are just working fine:
> 
> 
> # horus /srv/samba/demoshare # samba_dnsupdate --verbose
> --all-names # IPs:
> ['192.168.41.25'] # force update: A horus.samdom.com
> 192.168.41.25 # force update: NS samdom.com
> horus.samdom.com # force update: NS _msdcs.samdom.com
> horus.samdom.com # force update: A samdom.com
> 192.168.41.25 # .....
> # 29 DNS updates and 0 DNS deletes
> needed # Successfully obtained Kerberos ticket to
> DNS/charon.samdom.com as HORUS$ # update(nsupdate): A
> horus.samdom.com 192.168.41.25 # Calling nsupdate for A
> horus.samdom.com 192.168.41.25 (add) # Successfully obtained Kerberos
> ticket to DNS/charon.samdom.com as HORUS$ # Outgoing update
> query: # ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:
> 0 # ;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL:
> 0 # ;; UPDATE
> SECTION: # horus.samdom.com.    900     IN      A
> 192.168.41.25
> # # update(nsupdate): NS samdom.com
> horus.samdom.com # .....
> 
> 
> Now I edit resolv.conf to point to the ADDC charon at 192.168.11.205, 
> and the Kerberos ticket is now obtained by DNS/horus.samdom.com,
> which is actually on of the Gentoo machines, and even though it
> states the Ticket was granted successfully, the update fails.
> 
> 
> # horus ~ # samba_dnsupdate --verbose --all-names
> # IPs: ['192.168.41.25']
> # force update: A horus.samdom.com 192.168.41.25
> # .....
> # 29 DNS updates and 0 DNS deletes needed
> # Successfully obtained Kerberos ticket to DNS/horus.samdom.com as
> HORUS$ # update(nsupdate): A horus.samdom.com 192.168.41.25
> # Calling nsupdate for A horus.samdom.com 192.168.41.25 (add)
> # Successfully obtained Kerberos ticket to DNS/horus.samdom.com as
> HORUS$ # Outgoing update query:
> # ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
> # ;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
> # ;; UPDATE SECTION:
> # horus.samdom.com.    900     IN      A       192.168.41.25
> # 
> # dns_tkey_gssnegotiate: TKEY is unacceptable
> # Failed nsupdate: 1
> # update(nsupdate): NS samdom.com horus.samdom.com
> # .....
> 
> 
> Needless to say, that tried to generate new keytabs. I demoted
> machines and re-joined them, but the issue persists. Actually there
> is samba-4.8.3 on all machines, and the ldb/tdb/tevent/talloc in the
> same version as bundled with samba-4.8.3. Raspbian has a pretty old
> Bind  9.10.3-P4. On Gentoo I tried 9.11.3 and 9.11.2_p1. 
> 
> What I need first is a tip for an efficient setting for debugging it. 
> Is there a way to have a look on the granted tickets? There must be 
> some difference. 
> 
I think you have run into the 'whoever creates the dns records owns
them' problem. Only the owner of a dns record can update that record
and if you look carefully, you are trying to update the same records
from both machines. Try pointing the /etc/resolv.conf nameserver on
each DC to itself. If all else fails, you could also try adding
'--use-samba-tool' to the command.

Rowland

On Sat, 30 Jun 2018 18:08:22 +0200 (CEST)
"Peter Serbe" <peter at serbe.ch> wrote:
> 
> 
> Rowland Penny via samba schrieb am 30.06.2018 16:45:
> 
> > I think you have run into the 'whoever creates the dns records
owns
> > them' problem. 
> 
> Hi Rowland, 
> 
> I am extremely surprised by that. Writing the two lines below each
> other...
> 
> >> Successfully obtained Kerberos ticket to DNS/horus.home.serbe.ch
> >> as HORUS$ Successfully obtained Kerberos ticket to
> >> DNS/charon.home.serbe.ch as HORUS$
Yes, but only one of the machines can update the records, the other
will always fail.


 
> ... then I see, that there are different principals, and apparently
> the tickets on the machines are issued to the different principals.
> OK, so understand, that on one machine all the DNS entries must be
> owned by the principal, which is listed in the local keytab file,
> right? 
Yes, each machine can update its own records.
> 
> So the first question is: how can I make the local DNS to send out 
> the local machine as the first service provider. Currently it looks
> like there was a big mess. Every DNS spits out a different order...
> it should at least give out the own name before the others. 
Not sure I understand what you are trying to ask here, each dns server
is authoritative for the dns domain (multi-master), there is no single
master (unless you only have on DC) and there are definitely no slave
dns servers. Each DC should just sit there, awaiting the clients
asking for dns info.
 
> 
> Another think, which surprises me, is that this effect eats up a
> whole lot of the redundancy of the whole network. As the going down
> of one DNS would seriously disturb the capabilities of the DCs. But
> OK, it won't bring it down too fast, but - beware - one has to
> monitor the stuff.
As each DC is a dns master, this shouldn't be a problem, provide the
clients get the full set of nameservers.
 
> 
> Is there any Wiki-article discussing the issue? Or any blog post?
> The issue should be of major concern for any admin, who runs (as 
> advised by the Samba team) several ADDCs in one network. 
> 
> 
> > Only the owner of a dns record can update that record
> > and if you look carefully, you are trying to update the same records
> > from both machines. Try pointing the /etc/resolv.conf nameserver on
> > each DC to itself. 
> 
> I will do so, as soon as I understand the big picture - and of 
> course the means to get there:
Do you use dhcp for the clients ?
If so, there is a wikipage about running the dhcp server on a DC, see
here:

https://wiki.samba.org/index.php/Configure_DHCP_to_update_DNS_records_with_BIND9

There is also a page about bind9:

https://wiki.samba.org/index.php/Setting_up_a_BIND_DNS_Server
> - how to see the ownership of the DNS records. and in a second step
> - how to transfer the ownership of them
> - then: what happens with the DNS records ownership during the 
>   process of the replication over to the other ADDCs. Or maybe
>   this is even a no-problem... I am still a bit confused.
The record ownerships is stored in AD, you need to see the
'nTSecurityDescriptor' attribute of the dns record.

Rowland

Am Samstag, 30. Juni 2018, 16:01:10 CEST schrieb Peter Serbe via
samba:
> Dear Samba experts,
> 
> Since a couple of days I am trying to fix my domain.
> I have each two ADDCs on raspis on two sites. One is running on
> Raspian and works fine. The other three are on Gentoo and something
> is broken there. When I point the name resolution in resolv.conf to
> the Raspian machine the dynamic updates are just working fine:
> 
> 
> # horus /srv/samba/demoshare # samba_dnsupdate --verbose --all-
names
> # IPs: ['192.168.41.25']
> # force update: A horus.samdom.com 192.168.41.25
> # force update: NS samdom.com horus.samdom.com
> # force update: NS _msdcs.samdom.com horus.samdom.com
> # force update: A samdom.com 192.168.41.25
> # .....
> # 29 DNS updates and 0 DNS deletes needed
> # Successfully obtained Kerberos ticket to DNS/charon.samdom.com as
> HORUS$ # update(nsupdate): A horus.samdom.com 192.168.41.25
> # Calling nsupdate for A horus.samdom.com 192.168.41.25 (add)
> # Successfully obtained Kerberos ticket to DNS/charon.samdom.com as
> HORUS$ # Outgoing update query:
> # ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
> # ;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
> # ;; UPDATE SECTION:
> # horus.samdom.com.    900     IN      A       192.168.41.25
> #
> # update(nsupdate): NS samdom.com horus.samdom.com
> # .....
> 
> 
> Now I edit resolv.conf to point to the ADDC charon at 192.168.11.205,
> and the Kerberos ticket is now obtained by DNS/horus.samdom.com, 
which
> is actually on of the Gentoo machines, and even though it states the
> Ticket was granted successfully, the update fails.
> 
> 
> # horus ~ # samba_dnsupdate --verbose --all-names
> # IPs: ['192.168.41.25']
> # force update: A horus.samdom.com 192.168.41.25
> # .....
> # 29 DNS updates and 0 DNS deletes needed
> # Successfully obtained Kerberos ticket to DNS/horus.samdom.com as
> HORUS$ # update(nsupdate): A horus.samdom.com 192.168.41.25
> # Calling nsupdate for A horus.samdom.com 192.168.41.25 (add)
> # Successfully obtained Kerberos ticket to DNS/horus.samdom.com as
> HORUS$ # Outgoing update query:
> # ;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
> # ;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
> # ;; UPDATE SECTION:
> # horus.samdom.com.    900     IN      A       192.168.41.25
> #
> # dns_tkey_gssnegotiate: TKEY is unacceptable
> # Failed nsupdate: 1
> # update(nsupdate): NS samdom.com horus.samdom.com
> # .....
> 
> 
> Needless to say, that tried to generate new keytabs. I demoted
> machines and re-joined them, but the issue persists. Actually there
> is samba-4.8.3 on all machines, and the ldb/tdb/tevent/talloc in the
> same version as bundled with samba-4.8.3. Raspbian has a pretty old
> Bind  9.10.3-P4. On Gentoo I tried 9.11.3 and 9.11.2_p1.
> 
> What I need first is a tip for an efficient setting for debugging it.
> Is there a way to have a look on the granted tickets? There must be
> some difference.
> 
> I examined the output from named, but I could not see something fishy
> there.
> 
> This one works OK:
> > root at charon:/usr/local/samba/private# named -V
> > BIND 9.10.3-P4-Raspbian <id:ebd72b3>
> > built by make with '--prefix=/usr'
'--mandir=/usr/share/man'
> > '--libdir=/usr/lib/arm-linux-gnueabihf'
'--infodir=/usr/share/info'
> > '--sysconfdir=/etc/bind' '--with-python=python3'
'--localstatedir=/'
> > '--enable-threads' '--enable-largefile'
'--with-libtool'
> > '--enable-shared' '--enable-static'
'--with-gost=no'
> > '--with-openssl=/usr' '--with-gssapi=/usr'
'--with-gnu-ld'
> > '--with-geoip=/usr' '--with-atf=no'
'--enable-ipv6' '--enable-rrl'
> > '--enable-filter-aaaa' '--enable-native-pkcs11'
> >
'--with-pkcs11=/usr/lib/arm-linux-gnueabihf/softhsm/libsofthsm2.so'
> > '--with-randomdev=/dev/urandom' 'CFLAGS=-g -O2
> > -fdebug-prefix-map=/build/bind9-6GG44j/bind9-9.10.3.dfsg.P4=.
> > -fstack-protector-strong -Wformat -Werror=format-security
> > -fno-strict-aliasing -fno-delete-null-pointer-checks
> > -DNO_VERSION_DATE -DDIG_SIGCHASE' 'LDFLAGS=-Wl,-z,relro
-Wl,-z,now'
> > 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
Maybe --with-dlz-ldap is the default
> And this is not working (on the Gentoo machine):
> > horus /etc/portage # named -V
> > BIND 9.11.3 (Extended Support Version) <id:a375815>
> > running on Linux armv7l 
4.4.136-695e41116993e0a4f080354e72f13d91-0
> > #1
> > SMP Thu Jun 14 14:09:46 CEST 2018
> > built by make with '--prefix=/usr'
> > '--build=armv7a-hardfloat-linux-gnueabi'
> > '--host=armv7a-hardfloat-linux-gnueabi'
'--mandir=/usr/share/man'
> > '--infodir=/usr/share/info' '--datadir=/usr/share'
> > '--sysconfdir=/etc' '--localstatedir=/var/lib'
'--libdir=/usr/lib'
> > '--sysconfdir=/etc/bind' '--localstatedir=/var'
'--with-libtool'
> > '--enable-full-report' '--without-readline'
'--enable-linux-caps'
> > '--disable-filter-aaaa' '--disable-fixed-rrset'
'--disable-ipv6'
> > '--disable-rpz-nsdname' '--disable-rpz-nsip'
'--disable-seccomp'
> > '--enable-threads'
> > '--with-dlz-bdb' '--with-dlopen'
'--with-dlz-filesystem'
> > '--with-dlz-stub' '--with-gost'
'--with-gssapi' '--without-idn'
> > '--without-libjson' '--without-dlz-ldap'
'--without-dlz-mysql'
> > '--without-dlz-odbc' '--without-dlz-postgres'
'--without-lmdb'
> > '--with-python' '--with-ecdsa'
'--with-openssl=/usr'
> > '--without-libxml2' '--with-zlib'
> > '--with-randomdev=/dev/random'
> > 'build_alias=armv7a-hardfloat-linux-gnueabi'
> > 'host_alias=armv7a-hardfloat-linux-gnueabi' 'CFLAGS=-O2
-pipe
> > -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard
> > -I/usr/include/db5.3' 'LDFLAGS=-Wl,-O1 -Wl,--as-needed'
> > compiled by GCC 6.4.0
> > compiled with OpenSSL version: OpenSSL 1.0.2o  27 Mar 2018
> > linked to OpenSSL version: OpenSSL 1.0.2o  27 Mar 2018
> > compiled with zlib version: 1.2.11
> > linked to zlib version: 1.2.11
> > threads support is enabled
--with-dlz-ldap is diabled
 
> Thank You in advance and best regards
> Peter

-- 

Gruss
	Harry Jede