samba - Feb 2020 - Samba 4.11.6 cannot JOIN - 'Could not find machine account'

Hi Rowland,
> Can you try the join command like this:
> samba-tool domain join office.example.com DC -UAdministrator 
> --password=TheActualPassword --dns-backend=BIND9_DLZ
> Rowland
When I run samba-tool like this without specifying the server, it chooses the
older backup
server that runs Server 2008 (named PE2600).

Joining to this server results in a different error:

....
INFO 2020-02-10 19:27:18,369 pid:2400
/usr/lib/python3/dist-packages/samba/join.py #107:
 Finding a writeable DC for domain 'office.example.com'
resolve_lmhosts: Attempting lmhosts lookup for name
_ldap._tcp.office.example.com<0x0>
INFO 2020-02-10 19:27:18,387 pid:2400
/usr/lib/python3/dist-packages/samba/join.py #109:
 Found DC PE2600.office.example.com
resolve_lmhosts: Attempting lmhosts lookup for name
PE2600.office.example.com<0x20>
INFO 2020-02-10 19:27:24,690 pid:2400
/usr/lib/python3/dist-packages/samba/join.py #1542:
 workgroup is OFFICE
INFO 2020-02-10 19:27:24,691 pid:2400
/usr/lib/python3/dist-packages/samba/join.py #1545:
 realm is office.example.com
Using binding ncacn_ip_tcp:PE2600.office.example.com[,seal]
resolve_lmhosts: Attempting lmhosts lookup for name
PE2600.office.example.com<0x20>
resolve_lmhosts: Attempting lmhosts lookup for name
PE2600.office.example.com<0x20>
tdb(/var/lib/samba/private/secrets.tdb): tdb_open_ex: could not open file 
/var/lib/samba/private/secrets.tdb:
 No such file or directory
Could not open tdb: No such file or directory
ldb_wrap open of secrets.ldb
Could not find machine account in secrets database: Failed to fetch machine
account
password from secrets.ldb:
 Could not find entry to match filter:
'(&(flatname=OFFICE)(objectclass=primaryDomain))'
base: 'cn=Primary Domains':
  No such object: dsdb_search at ../../source4/dsdb/common/util.c:4733
   and failed to open /var/lib/samba/private/secrets.tdb: 
NT_STATUS_CANT_ACCESS_DOMAIN_INFO
ERROR(ldb): uncaught exception - LDAP error 10 LDAP_REFERRAL -  <0000202B:
RefErr:
DSID-030A0AEB, data 0, 1 access points
        ref 1:
'4da1d2ff-c0a3-45f8-8e4e-c3dcce17473c._msdcs.office.example.com'
>
<ldap://4da1d2ff-c0a3-45f8-8e4e-c3dcce17473c._msdcs.office.example.com>
  File "/usr/lib/python3/dist-packages/samba/netcmd/__init__.py", line
186, in _run
    return self.run(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/samba/netcmd/domain.py", line
708, in run
    backend_store_size=backend_store_size)
  File "/usr/lib/python3/dist-packages/samba/join.py", line 1558, in
join_DC
    ctx.do_join()
  File "/usr/lib/python3/dist-packages/samba/join.py", line 1446, in
do_join
    ctx.join_add_objects()
  File "/usr/lib/python3/dist-packages/samba/join.py", line 711, in
join_add_objects
    ctx.samdb.modify(m)
Adding CN=SAMBA1,OU=Domain Controllers,DC=office,DC=example,DC=com
Adding 
CN=SAMBA1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=offi
ce,DC=example,DC=com
Adding CN=NTDS 
Settings,CN=SAMBA1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuratio
n,DC=office,DC=example,DC=com
Join failed - cleaning up
Deleted CN=SAMBA1,OU=Domain Controllers,DC=office,DC=example,DC=com
Deleted CN=NTDS 
Settings,CN=SAMBA1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuratio
n,DC=office,DC=example,DC=com
Deleted 
CN=SAMBA1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=offi
ce,DC=example,DC=com
-----

FWIW, '4da1d2ff-c0a3-45f8-8e4e-c3dcce17473c._msdcs.office.example.com'
IS the correct
GUID for the primary server, SERVI, (Server 2008 R2), so not sure why this error
is thrown.

I looked into this default server choice by samba-tool over on the Windows
server side.
I saw that both PE2600 and SERVI entries in the _ldap entries had Priority=0
(highest).
Tweaking the Priority for PE2600 from 0 to 1 and forcing replication now helps
samba-tool
automatically pick the R2 server, SERVI, instead of PE2600.

So, running samba-tool without "--server" now produces the original
error

...
ERROR(runtime): uncaught exception - 
(9003,'WERR_DNS_ERROR_RCODE_NAME_ERROR')
  File "/usr/lib/python3/dist-packages/samba/netcmd/__init__.py", line
186, in _run
    return self.run(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/samba/netcmd/domain.py", line
708, in run
    backend_store_size=backend_store_size)
  File "/usr/lib/python3/dist-packages/samba/join.py", line 1558, in
join_DC
    ctx.do_join()
  File "/usr/lib/python3/dist-packages/samba/join.py", line 1455, in
do_join
    ctx.join_add_dns_records()
  File "/usr/lib/python3/dist-packages/samba/join.py", line 1197, in
join_add_dns_records
    dns_partition=domaindns_zone_dn)
  File "/usr/lib/python3/dist-packages/samba/samdb.py", line 1177, in
dns_lookup
    dns_partition=dns_partition)
-------

So, no progress yet :-(

I keep looking for any problem in the AD contents and the Windows Event logs,
but am still
lacking a solution.

If you can think of any more detailed logging I can do to see what Samba is
doing here, I will
dig deeper!

Thanks for your help on this mystery!