Michael Tokarev
2022-Nov-16 07:33 UTC
[Samba] Replication between Samba DCs (on different sites)?
Replying to my own emails and thread.. 15.11.2022 00:07, Michael Tokarev via samba wrote:> 14.11.2022 23:21, Michael Tokarev via samba wrote: > ... >> I tried 'samba-tool drs replicate' manually on AI, but it also shows this >> error: >> >> AI# samba-tool drs replicate ai svdcp 'CN=Configuration,DC=tls,DC=msk,DC=ru' >> ERROR(<class 'samba.drs_utils.drsException'>): DsReplicaSync failed - drsException: DsReplicaSync failed (2, 'WERR_FILE_NOT_FOUND') >> ?? File "/usr/lib/python3/dist-packages/samba/netcmd/drs.py", line 570, in run >> ???? drs_utils.sendDsReplicaSync(server_bind, server_bind_handle, source_dsa_guid, NC, req_options) >> ?? File "/usr/lib/python3/dist-packages/samba/drs_utils.py", line 92, in sendDsReplicaSync >> ???? raise drsException("DsReplicaSync failed %s" % estr) > > > AI# samba-tool drs replicate ai svdcp 'CN=Configuration,DC=tls,DC=msk,DC=ru' -d 10 > > gives some wire traces (or looks like), it ends up like the remote is returning > WERR_FILE_NOT_FOUND. And I don't see this error on the remote, all what I see > remote reporting on the logs is WERR_OK.So, after recompiling samba multiple times adding numerous debugging messages into *_DsReplicaSync and below, I found out the database on the "primary" DC contained a few references to the objects I had to remove before, for example: NOTE: old (due to rename or delete) DN string component for rIDSetReferences in object CN=SVDCM\0ADEL:a1a97bca-fbdf-429a-966e-cb8d71da606c,CN=Deleted Objects,DC=tls,DC=msk,DC=ru - CN=RID Set,CN=SVDCM,OU=Domain Controllers,DC=tls,DC=msk,DC=ru (note the CN=Deleted Objects). It was a long and painful debugging which lasted 2 complete days. After all this, when trying to find a way to get a dump of ldb - I found (by a chance) samba-tool dbcheck. Which found all these objects (but displayed "0 errors" anyway). And after removing these "Deleted Objects" things, it started working fine. There are just 329 objects in the db now. So, basically, samba-tool dbcheck for the rescue at the very least, and note that renames/deletes in samba does not quite work. Thank everyone for the help, /mjt
Lorenzo Milesi
2023-Feb-07 18:19 UTC
[Samba] Replication between Samba DCs (on different sites)?
>> gives some wire traces (or looks like), it ends up like the remote is returning >> WERR_FILE_NOT_FOUND. And I don't see this error on the remote, all what I see >> remote reporting on the logs is WERR_OK. > > So, after recompiling samba multiple times adding numerous debugging messages > into *_DsReplicaSync and below, I found out the database on the "primary" DC > contained a few references to the objects I had to remove before, for example: > > NOTE: old (due to rename or delete) DN string component for rIDSetReferences in > object CN=SVDCM\0ADEL:a1a97bca-fbdf-429a-966e-cb8d71da606c,CN=Deleted > Objects,DC=tls,DC=msk,DC=ru - CN=RID Set,CN=SVDCM,OU=Domain > Controllers,DC=tls,DC=msk,DC=ru > > (note the CN=Deleted Objects). > > It was a long and painful debugging which lasted 2 complete days. > > After all this, when trying to find a way to get a dump of ldb - I found > (by a chance) samba-tool dbcheck. Which found all these objects (but > displayed "0 errors" anyway). And after removing these "Deleted Objects" > things, it started working fine. > > There are just 329 objects in the db now. > > So, basically, samba-tool dbcheck for the rescue at the very least, > and note that renames/deletes in samba does not quite work.I'm stuck in a similar situation. I've upgraded a Samba4 network (3 DCs) from 4.13 to 4.17 using demote/join. Now dc1 and dc3 are fine, while dc2 is reporting WERR_FILE_NOT_FOUND on all sync items. I ran samba-tool dbcheck and samba-tool dbcheck --cross-ncs several times, no errors are reported (anymore). But whenever I attempt a replication, either of the full domain or of a single item, I get: ERROR(<class 'samba.drs_utils.drsException'>): DsReplicaSync failed - drsException: DsReplicaSync failed (2, 'WERR_FILE_NOT_FOUND') File "/usr/lib/python3/dist-packages/samba/netcmd/drs.py", line 570, in run drs_utils.sendDsReplicaSync(server_bind, server_bind_handle, source_dsa_guid, NC, req_options) File "/usr/lib/python3/dist-packages/samba/drs_utils.py", line 100, in sendDsReplicaSync raise drsException("DsReplicaSync failed %s" % estr) `samba-tool drs showrepl` on dc1 shows: root at dc1:/var/lib/samba/bind-dns/dns# samba-tool drs showrepl Default-First-Site-Name\DC1 DSA Options: 0x00000001 DSA object GUID: 5b1453be-b074-4d33-9169-796f85eed444 DSA invocationId: e6cb3930-897e-4ba9-952d-28802ace401d ==== INBOUND NEIGHBORS === CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC2 via RPC DSA object GUID: 39a77331-7665-49bf-8dd4-89e19a1b1709 Last attempt @ Tue Feb 7 19:01:09 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 19:01:09 2023 CET CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ Tue Feb 7 19:01:09 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 19:01:09 2023 CET DC=DomainDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC2 via RPC DSA object GUID: 39a77331-7665-49bf-8dd4-89e19a1b1709 Last attempt @ Tue Feb 7 19:01:08 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 19:01:08 2023 CET DC=DomainDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ Tue Feb 7 19:01:08 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 19:01:08 2023 CET CN=Schema,CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC2 via RPC DSA object GUID: 39a77331-7665-49bf-8dd4-89e19a1b1709 Last attempt @ Tue Feb 7 19:01:09 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 19:01:09 2023 CET CN=Schema,CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ Tue Feb 7 19:01:09 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 19:01:09 2023 CET DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC2 via RPC DSA object GUID: 39a77331-7665-49bf-8dd4-89e19a1b1709 Last attempt @ Tue Feb 7 19:01:09 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 19:01:09 2023 CET DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ Tue Feb 7 19:01:09 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 19:01:09 2023 CET DC=ForestDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC2 via RPC DSA object GUID: 39a77331-7665-49bf-8dd4-89e19a1b1709 Last attempt @ Tue Feb 7 19:01:08 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 19:01:08 2023 CET DC=ForestDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ Tue Feb 7 19:01:09 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 19:01:09 2023 CET ==== OUTBOUND NEIGHBORS === CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ NTTIME(0) was successful 0 consecutive failure(s). Last success @ NTTIME(0) DC=DomainDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ NTTIME(0) was successful 0 consecutive failure(s). Last success @ NTTIME(0) CN=Schema,CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ NTTIME(0) was successful 0 consecutive failure(s). Last success @ NTTIME(0) DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ NTTIME(0) was successful 0 consecutive failure(s). Last success @ NTTIME(0) DC=ForestDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ NTTIME(0) was successful 0 consecutive failure(s). Last success @ NTTIME(0) ==== KCC CONNECTION OBJECTS === Connection -- Connection name: b058383e-10b2-4d00-a87d-30f88dd41db3 Enabled : TRUE Server DNS name : dc2.wdc.domain.it Server DN name : CN=NTDS Settings,CN=DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=wdc,DC=domain,DC=it TransportType: RPC options: 0x00000001 Warning: No NC replicated for Connection! Connection -- Connection name: e5bbfdc3-9b2f-4bdb-82a5-52d2a3b73d9f Enabled : TRUE Server DNS name : exmedc.wdc.domain.it Server DN name : CN=NTDS Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=wdc,DC=domain,DC=it TransportType: RPC options: 0x00000001 Warning: No NC replicated for Connection! While on (failing) dc2: root at dc2:/var/lib/samba/bind-dns# samba-tool drs showrepl Default-First-Site-Name\DC2 DSA Options: 0x00000001 DSA object GUID: 39a77331-7665-49bf-8dd4-89e19a1b1709 DSA invocationId: c39e6ca3-e46d-4994-be0a-6aa647b6934b ==== INBOUND NEIGHBORS === CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ Tue Feb 7 18:59:10 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 18:59:10 2023 CET CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC1 via RPC DSA object GUID: 5b1453be-b074-4d33-9169-796f85eed444 Last attempt @ Tue Feb 7 19:00:14 2023 CET failed, result 2 (WERR_FILE_NOT_FOUND) 18 consecutive failure(s). Last success @ NTTIME(0) DC=DomainDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ Tue Feb 7 19:05:10 2023 CET failed, result 121 (WERR_SEM_TIMEOUT) 1 consecutive failure(s). Last success @ Tue Feb 7 18:59:10 2023 CET DC=DomainDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC1 via RPC DSA object GUID: 5b1453be-b074-4d33-9169-796f85eed444 Last attempt @ Tue Feb 7 19:05:10 2023 CET failed, result 2 (WERR_FILE_NOT_FOUND) 14 consecutive failure(s). Last success @ NTTIME(0) CN=Schema,CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ Tue Feb 7 18:59:10 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 18:59:10 2023 CET CN=Schema,CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC1 via RPC DSA object GUID: 5b1453be-b074-4d33-9169-796f85eed444 Last attempt @ Tue Feb 7 18:59:10 2023 CET failed, result 2 (WERR_FILE_NOT_FOUND) 13 consecutive failure(s). Last success @ NTTIME(0) DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ Tue Feb 7 18:59:11 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 18:59:11 2023 CET DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC1 via RPC DSA object GUID: 5b1453be-b074-4d33-9169-796f85eed444 Last attempt @ Tue Feb 7 19:00:25 2023 CET failed, result 2 (WERR_FILE_NOT_FOUND) 20 consecutive failure(s). Last success @ NTTIME(0) DC=ForestDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ Tue Feb 7 18:59:10 2023 CET was successful 0 consecutive failure(s). Last success @ Tue Feb 7 18:59:10 2023 CET DC=ForestDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC1 via RPC DSA object GUID: 5b1453be-b074-4d33-9169-796f85eed444 Last attempt @ Tue Feb 7 18:59:10 2023 CET failed, result 2 (WERR_FILE_NOT_FOUND) 13 consecutive failure(s). Last success @ NTTIME(0) ==== OUTBOUND NEIGHBORS === CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC1 via RPC DSA object GUID: 5b1453be-b074-4d33-9169-796f85eed444 Last attempt @ Tue Feb 7 19:04:06 2023 CET failed, result 2 (WERR_FILE_NOT_FOUND) 36 consecutive failure(s). Last success @ NTTIME(0) CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ NTTIME(0) was successful 0 consecutive failure(s). Last success @ NTTIME(0) DC=DomainDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC1 via RPC DSA object GUID: 5b1453be-b074-4d33-9169-796f85eed444 Last attempt @ Tue Feb 7 19:04:06 2023 CET failed, result 2 (WERR_FILE_NOT_FOUND) 36 consecutive failure(s). Last success @ NTTIME(0) DC=DomainDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ NTTIME(0) was successful 0 consecutive failure(s). Last success @ NTTIME(0) CN=Schema,CN=Configuration,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC1 via RPC DSA object GUID: 5b1453be-b074-4d33-9169-796f85eed444 Last attempt @ Tue Feb 7 19:04:06 2023 CET failed, result 2 (WERR_FILE_NOT_FOUND) 36 consecutive failure(s). Last success @ NTTIME(0) CN=Schema,CN=Configuration,DC=wdc,DC=domain,DC=it [13/1937] Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ NTTIME(0) was successful 0 consecutive failure(s). Last success @ NTTIME(0) DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC1 via RPC DSA object GUID: 5b1453be-b074-4d33-9169-796f85eed444 Last attempt @ Tue Feb 7 19:04:06 2023 CET failed, result 2 (WERR_FILE_NOT_FOUND) 36 consecutive failure(s). Last success @ NTTIME(0) DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ NTTIME(0) was successful 0 consecutive failure(s). Last success @ NTTIME(0) DC=ForestDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC1 via RPC DSA object GUID: 5b1453be-b074-4d33-9169-796f85eed444 Last attempt @ Tue Feb 7 19:04:06 2023 CET failed, result 2 (WERR_FILE_NOT_FOUND) 36 consecutive failure(s). Last success @ NTTIME(0) DC=ForestDnsZones,DC=wdc,DC=domain,DC=it Default-First-Site-Name\DC3 via RPC DSA object GUID: c3dcc23e-d2b3-4899-a615-3f4559b3a647 Last attempt @ NTTIME(0) was successful 0 consecutive failure(s). Last success @ NTTIME(0) ==== KCC CONNECTION OBJECTS === Connection -- Connection name: ef26d1b1-771e-4feb-9049-e2dbf9ab6f64 Enabled : TRUE Server DNS name : dc1.wdc.domain.it Server DN name : CN=NTDS Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=wdc,DC=domain,DC=it TransportType: RPC options: 0x00000001 Warning: No NC replicated for Connection! Connection -- Connection name: 219691ab-12bd-49ab-8a92-0570cabb3589 Enabled : TRUE Server DNS name : exmedc.wdc.domain.it Server DN name : CN=NTDS Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=wdc,DC=domain,DC=it TransportType: RPC options: 0x00000001 Warning: No NC replicated for Connection! There must be some stale record but I have no idea how to dig them out. One ODD thing I found via RSAT: dc2 used to be into a separate site, while with DC demotion and rejoin has fallen into "Default-First-Site-Name". But Active Directory Sites and Services shows TWO DC2 records, one in the aforementioned site and one in the "original" one. The "original" one doesn't have a "NTDS Settings" child entry into the tree. I deleted it but apparently didn't have any impact. I moved DC2 back to the original site. DC2 is also now hodling all FSMO roles, and if I try to take them back on DC1: #?samba-tool fsmo transfer --role=all -U administrator ERROR: Transfer of 'rid' role failed: Failed FSMO transfer: WERR_NETNAME_DELETED A second attempt transferred some of the roles, but still resulted in an error: root at dc1:/var/lib/samba/bind-dns/dns# samba-tool fsmo transfer --role=all -U administrator This DC already has the 'rid' FSMO role This DC already has the 'pdc' FSMO role This DC already has the 'naming' FSMO role This DC already has the 'infrastructure' FSMO role This DC already has the 'schema' FSMO role Password for [WDC\administrator]: ERROR: Failed to add role 'domaindns': LDAP error 16 LDAP_NO_SUCH_ATTRIBUTE - <attribute 'fSMORoleOwner': no matching attribute value while deleting attribute on 'CN=Infrastructure,DC=DomainDnsZones,DC=wdc,DC=domain,DC=it'> <> Now all roles except DomainDnsZonesMasterRole and ForestDnsZonesMasterRole shows as residing on DC1: root at dc1:/var/lib/samba/bind-dns/dns# samba-tool fsmo show SchemaMasterRole owner: CN=NTDS Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=wdc,DC=domain,DC=it InfrastructureMasterRole owner: CN=NTDS Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=wdc,DC=domain,DC=it RidAllocationMasterRole owner: CN=NTDS Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=wdc,DC=domain,DC=it PdcEmulationMasterRole owner: CN=NTDS Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=wdc,DC=domain,DC=it DomainNamingMasterRole owner: CN=NTDS Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=wdc,DC=domain,DC=it DomainDnsZonesMasterRole owner: CN=NTDS Settings,CN=DC2,CN=Servers,CN=ARUBA-DataCenter1,CN=Sites,CN=Configuration,DC=wdc,DC=domain,DC=it ForestDnsZonesMasterRole owner: CN=NTDS Settings,CN=DC2,CN=Servers,CN=ARUBA-DataCenter1,CN=Sites,CN=Configuration,DC=wdc,DC=domain,DC=it All servers run 4.17.5-Ubuntu (from mjt). If matters: root at dc1:/var/lib/samba/bind-dns/dns# samba-tool domain level show Domain and forest function level for domain 'DC=wdc,DC=domain,DC=it' Forest function level: (Windows) 2008 R2 Domain function level: (Windows) 2008 R2 Lowest function level of a DC: (Windows) 2008 R2 Thanks -- Lorenzo Milesi - lorenzo.milesi at yetopen.com CTO @ YetOpen Srl Corso Martiri della Liberazione 114 - 23900 Lecco - ITALY - | 4801 Glenwood Avenue - Suite 200 - Raleigh, NC 27612 - USA - Tel +39 0341 220 205 - info.it at yetopen.com | Phone +1 919-817-8106 - info.us at yetopen.com Think green - Non stampare questa e-mail se non necessario / Don't print this email unless necessary -------- D.Lgs. 196/2003 e GDPR 679/2016 -------- Tutte le informazioni contenute in questo messaggio sono riservate ed a uso esclusivo del destinatario. Tutte le informazioni ivi contenute, compresi eventuali allegati, sono da ritenere confidenziali e riservate secondo i termini del vigente D.Lgs. 196/2003 in materia di privacy e del Regolamento europeo 679/2016 - GDPR - e quindi ne e' proibita l'utilizzazione ulteriore non autorizzata. Nel caso in cui questo messaggio Le fosse pervenuto per errore, La invitiamo ad eliminarlo senza copiarlo, stamparlo, a non inoltrarlo a terzi e ad avvertirci non appena possibile. Grazie. Confidentiality notice: this email message including any attachment is for the sole use of the intended recipient and may contain confidential and privileged information; pursuant to Legislative Decree 196/2003 and the European General Data Protection Regulation 679/2016 - GDPR - any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recepient please delete this message without copying, printing or forwarding it to others, and alert us as soon as possible. Thank you.