samba - Dec 2019 - Replication not working for remote Domain Controller

Hi,
I have three Samba Domain Controllers, two in the LAN local network (dc1
and dc2) and one in a remote network which is accessible from the LAN
through a VPN connection (dc4).
Every domain controller can reach every other domain controllers, and every
type of traffic is permitted by firewalls, they can ping and access to
every TCP and UDP ports.

Checking the Samba replication I see that the two domain controllers on the
local network (dc1 and dc2) are regularly replicated, but dc4 is not
replicated at all:

================================== 8<
=========================================root at dc1:~/check_ad_replication.py
(18:59:47)# ./check_ad_replication.py
CRITICAL: Realm: tn.ies.it Failing: dc4 since forever(!!), Still OK: dc2 as
of 2 mins|ok=1 fail=1

root at dc2:~/check_ad_replication.py# ./check_ad_replication.py
CRITICAL: Realm: tn.ies.it Failing: dc4 since forever(!!), Still OK: dc1 as
of 1 mins|ok=1 fail=1
================================== 8<
=========================================
So I checked the replication status using "samba-tool drs showrepl"
and
it's clear that dc4 is not replicating, and I realized that I have
several WERR_FILE_NOT_FOUND errors for dc4 (see below).

I'm hanged trying to find out why I'm receiving the WERR_FILE_NOT_FOUND
error for dc4, so I checked the DNS with the Windows Active Directory Sites
and Services tool, and I saw that dc1 and dc2 both have two "replicate
from" connections, but dc4 has no connection.

In the Sites Subnets i only see the LAN network subnet and not the data
center one (the dc4's subnet): I don't know if this is a problem, but
it's
a difference.

The other difference I found in the Windows DNS tool is that there are no
records in the _msdcs.my.domain.name domain for dc4.

================================== 8<
=========================================root at dc1:/ (19:01:50)# samba-tool
drs showrepl
ldb_wrap open of secrets.ldb
GENSEC backend 'gssapi_spnego' registered
GENSEC backend 'gssapi_krb5' registered
GENSEC backend 'gssapi_krb5_sasl' registered
GENSEC backend 'spnego' registered
GENSEC backend 'schannel' registered
GENSEC backend 'naclrpc_as_system' registered
GENSEC backend 'sasl-EXTERNAL' registered
GENSEC backend 'ntlmssp' registered
GENSEC backend 'ntlmssp_resume_ccache' registered
GENSEC backend 'http_basic' registered
GENSEC backend 'http_ntlm' registered
GENSEC backend 'krb5' registered
GENSEC backend 'fake_gssapi_krb5' registered
Using binding ncacn_ip_tcp:dc1.my.domain.name[,seal]
resolve_lmhosts: Attempting lmhosts lookup for name
dc1.my.domain.name<0x20>
resolve_lmhosts: Attempting lmhosts lookup for name
dc1.my.domain.name<0x20>
resolve_lmhosts: Attempting lmhosts lookup for name
dc1.my.domain.name<0x20>
tn\DC1
DSA Options: 0x00000001
DSA object GUID: 419f9e5a-dc5d-4d03-9f8c-518b5d977b5c
DSA invocationId: 87154209-6015-40ff-b209-27482055eda8

==== INBOUND NEIGHBORS ===
DC=my,DC=domain,DC=name
tn\DC2 via RPC
DSA object GUID: 009ad456-64de-43a9-a321-dbdbb25fa21b
Last attempt @ Wed Dec 11 18:56:56 2019 CET was successful
0 consecutive failure(s).
Last success @ Wed Dec 11 18:56:56 2019 CET

DC=my,DC=domain,DC=name
tn\DC4 via RPC
DSA object GUID: d6106573-e676-49b7-b90a-7bdcd2ab4970
Last attempt @ Wed Dec 11 18:56:56 2019 CET failed, result 2
(WERR_FILE_NOT_FOUND) <========34 consecutive failure(s).
Last success @ NTTIME(0)

DC=ForestDnsZones,DC=my,DC=domain,DC=name
tn\DC2 via RPC
DSA object GUID: 009ad456-64de-43a9-a321-dbdbb25fa21b
Last attempt @ Wed Dec 11 19:00:53 2019 CET was successful
0 consecutive failure(s).
Last success @ Wed Dec 11 19:00:53 2019 CET

DC=ForestDnsZones,DC=my,DC=domain,DC=name
tn\DC4 via RPC
DSA object GUID: d6106573-e676-49b7-b90a-7bdcd2ab4970
Last attempt @ Wed Dec 11 18:56:55 2019 CET failed, result 2
(WERR_FILE_NOT_FOUND) <========34 consecutive failure(s).
Last success @ NTTIME(0)

DC=DomainDnsZones,DC=my,DC=domain,DC=name
tn\DC2 via RPC
DSA object GUID: 009ad456-64de-43a9-a321-dbdbb25fa21b
Last attempt @ Wed Dec 11 18:56:55 2019 CET was successful
0 consecutive failure(s).
Last success @ Wed Dec 11 18:56:55 2019 CET

DC=DomainDnsZones,DC=my,DC=domain,DC=name
tn\DC4 via RPC
DSA object GUID: d6106573-e676-49b7-b90a-7bdcd2ab4970
Last attempt @ Wed Dec 11 18:56:55 2019 CET failed, result 2
(WERR_FILE_NOT_FOUND) <========34 consecutive failure(s).
Last success @ NTTIME(0)

CN=Schema,CN=Configuration,DC=my,DC=domain,DC=name
tn\DC2 via RPC
DSA object GUID: 009ad456-64de-43a9-a321-dbdbb25fa21b
Last attempt @ Wed Dec 11 18:56:56 2019 CET was successful
0 consecutive failure(s).
Last success @ Wed Dec 11 18:56:56 2019 CET

CN=Schema,CN=Configuration,DC=my,DC=domain,DC=name
tn\DC4 via RPC
DSA object GUID: d6106573-e676-49b7-b90a-7bdcd2ab4970
Last attempt @ Wed Dec 11 18:56:56 2019 CET failed, result 2
(WERR_FILE_NOT_FOUND) <========34 consecutive failure(s).
Last success @ NTTIME(0)

CN=Configuration,DC=my,DC=domain,DC=name
tn\DC2 via RPC
DSA object GUID: 009ad456-64de-43a9-a321-dbdbb25fa21b
Last attempt @ Wed Dec 11 18:56:56 2019 CET was successful
0 consecutive failure(s).
Last success @ Wed Dec 11 18:56:56 2019 CET

CN=Configuration,DC=my,DC=domain,DC=name
tn\DC4 via RPC
DSA object GUID: d6106573-e676-49b7-b90a-7bdcd2ab4970
Last attempt @ Wed Dec 11 18:56:56 2019 CET failed, result 2
(WERR_FILE_NOT_FOUND) <========34 consecutive failure(s).
Last success @ NTTIME(0)

==== OUTBOUND NEIGHBORS ===
DC=my,DC=domain,DC=name
tn\DC2 via RPC
DSA object GUID: 009ad456-64de-43a9-a321-dbdbb25fa21b
Last attempt @ NTTIME(0) was successful
0 consecutive failure(s).
Last success @ NTTIME(0)

DC=my,DC=domain,DC=name
tn\DC4 via RPC
DSA object GUID: d6106573-e676-49b7-b90a-7bdcd2ab4970
Last attempt @ Wed Dec 11 19:01:48 2019 CET failed, result 2
(WERR_FILE_NOT_FOUND) <========3 consecutive failure(s).
Last success @ NTTIME(0)

DC=ForestDnsZones,DC=my,DC=domain,DC=name
tn\DC2 via RPC
DSA object GUID: 009ad456-64de-43a9-a321-dbdbb25fa21b
Last attempt @ NTTIME(0) was successful
0 consecutive failure(s).
Last success @ NTTIME(0)

DC=ForestDnsZones,DC=my,DC=domain,DC=name
tn\DC4 via RPC
DSA object GUID: d6106573-e676-49b7-b90a-7bdcd2ab4970
Last attempt @ Wed Dec 11 19:01:48 2019 CET failed, result 2
(WERR_FILE_NOT_FOUND) <========3 consecutive failure(s).
Last success @ NTTIME(0)

DC=DomainDnsZones,DC=my,DC=domain,DC=name
tn\DC2 via RPC
DSA object GUID: 009ad456-64de-43a9-a321-dbdbb25fa21b
Last attempt @ NTTIME(0) was successful
0 consecutive failure(s).
Last success @ NTTIME(0)

DC=DomainDnsZones,DC=my,DC=domain,DC=name
tn\DC4 via RPC
DSA object GUID: d6106573-e676-49b7-b90a-7bdcd2ab4970
Last attempt @ Wed Dec 11 19:01:48 2019 CET failed, result 2
(WERR_FILE_NOT_FOUND) <========3 consecutive failure(s).
Last success @ NTTIME(0)

CN=Schema,CN=Configuration,DC=my,DC=domain,DC=name
tn\DC2 via RPC
DSA object GUID: 009ad456-64de-43a9-a321-dbdbb25fa21b
Last attempt @ NTTIME(0) was successful
0 consecutive failure(s).
Last success @ NTTIME(0)

CN=Schema,CN=Configuration,DC=my,DC=domain,DC=name
tn\DC4 via RPC
DSA object GUID: d6106573-e676-49b7-b90a-7bdcd2ab4970
Last attempt @ Wed Dec 11 19:01:48 2019 CET failed, result 2
(WERR_FILE_NOT_FOUND) <========3 consecutive failure(s).
Last success @ NTTIME(0)

CN=Configuration,DC=my,DC=domain,DC=name
tn\DC2 via RPC
DSA object GUID: 009ad456-64de-43a9-a321-dbdbb25fa21b
Last attempt @ NTTIME(0) was successful
0 consecutive failure(s).
Last success @ NTTIME(0)

CN=Configuration,DC=my,DC=domain,DC=name
tn\DC4 via RPC
DSA object GUID: d6106573-e676-49b7-b90a-7bdcd2ab4970
Last attempt @ Wed Dec 11 19:01:48 2019 CET failed, result 2
(WERR_FILE_NOT_FOUND) <========3 consecutive failure(s).
Last success @ NTTIME(0)

==== KCC CONNECTION OBJECTS ===
Connection --
Connection name: b02289b7-419f-4c09-b2bf-914473d76731
Enabled        : TRUE
Server DNS name : dc2.my.domain.name
Server DN name  : CN=NTDS
Settings,CN=DC2,CN=Servers,CN=tn,CN=Sites,CN=Configuration,DC=my,DC=domain,DC=name
TransportType: RPC
options: 0x00000001
Warning: No NC replicated for Connection!
Connection --
Connection name: 91c5d42f-0e60-43b9-9a0c-e0b6dec70120
Enabled        : TRUE
Server DNS name : dc4.my.domain.name
Server DN name  : CN=NTDS
Settings,CN=DC4,CN=Servers,CN=tn,CN=Sites,CN=Configuration,DC=my,DC=domain,DC=name
TransportType: RPC
options: 0x00000001
Warning: No NC replicated for Connection!
================================== 8<
=========================================
Could you help me please?
Thank you very much!

On 11/12/2019 18:10, shacky via samba wrote:
> Hi,
> I have three Samba Domain Controllers, two in the LAN local network (dc1
> and dc2) and one in a remote network which is accessible from the LAN
> through a VPN connection (dc4).
> Every domain controller can reach every other domain controllers, and every
> type of traffic is permitted by firewalls, they can ping and access to
> every TCP and UDP ports.
>
> Checking the Samba replication I see that the two domain controllers on the
> local network (dc1 and dc2) are regularly replicated, but dc4 is not
> replicated at all:
>
> ================================== 8<
> =========================================> root at
dc1:~/check_ad_replication.py (18:59:47)# ./check_ad_replication.py
> CRITICAL: Realm: tn.ies.it Failing: dc4 since forever(!!), Still OK: dc2 as
> of 2 mins|ok=1 fail=1
>
> root at dc2:~/check_ad_replication.py# ./check_ad_replication.py
> CRITICAL: Realm: tn.ies.it Failing: dc4 since forever(!!), Still OK: dc1 as
> of 1 mins|ok=1 fail=1
> ================================== 8<
> =========================================>
> So I checked the replication status using "samba-tool drs
showrepl" and
> it's clear that dc4 is not replicating, and I realized that I have
> several WERR_FILE_NOT_FOUND errors for dc4 (see below).
>
> I'm hanged trying to find out why I'm receiving the
WERR_FILE_NOT_FOUND
> error for dc4, so I checked the DNS with the Windows Active Directory Sites
> and Services tool, and I saw that dc1 and dc2 both have two "replicate
> from" connections, but dc4 has no connection.
>
> In the Sites Subnets i only see the LAN network subnet and not the data
> center one (the dc4's subnet): I don't know if this is a problem,
but it's
> a difference.
>
> The other difference I found in the Windows DNS tool is that there are no
> records in the _msdcs.my.domain.name domain for dc4.
>
Good, the _msdcs domain is the forest domain, but are there records for 
all three DCs in:

DC=your.domain.com,CN=MicrosoftDNS,DC=DomainDnsZones,DC=your,DC=domain,DC=com

What version(s) of Samba is this ?

Rowland

>
> Good, the _msdcs domain is the forest domain

So is it normal that DC4 is not in that?

> but are there records for
all three DCs in:
>
> DC=your.domain.com
> ,CN=MicrosoftDNS,DC=DomainDnsZones,DC=your,DC=domain,DC=com
>
I tried to find this path in the LDAP Browser and ASDI Edit but I did not
manage in find it.
In the Windows DNS Manager connected to DC1 I found _ldap SRV records for
dc1 and dc2 into DNS\dc1\Forward Lookup Zones\my.domain.com
\domaindnszones\_sites\mysite\_tcp.
There are no record for dc4 there.

> What version(s) of Samba is this ?
>
Samba Version 4.6.7-Ubuntu on every three domain controllers.

Thanks!