samba - Apr 2019 - Replication failures

I’m getting errors with replication

Samba 4.9.3 on Ubuntu 16.04
6 Samba DC’s at 3 sites

I recently joined a Win2k8 R2 server to the domain and promoted it to domain
controller. The replication was working before that, and not working after.

I didn’t realize the problems and tried to join a Win2012 R2 server and promote
to DC as well. That also failed with an error about replication failures.


sudo samba-tool drs showrepl -d9
output from “showrepl”

==
clipped
.
.
.
GSSAPI Connection will be cryptographically signed
    drsuapi_DsReplicaGetInfo: struct drsuapi_DsReplicaGetInfo
       in: struct drsuapi_DsReplicaGetInfo
           bind_handle              : *
               bind_handle: struct policy_handle
                   handle_type              : 0x00000000 (0)
                   uuid                     :
cffaee4f-9caa-4e72-a89d-1d50df9dfa67
           level                    : DRSUAPI_DS_REPLICA_GET_INFO (1)
           req                      : *
               req                      : union
drsuapi_DsReplicaGetInfoRequest(case 1)
               req1: struct drsuapi_DsReplicaGetInfoRequest1
                   info_type                : DRSUAPI_DS_REPLICA_INFO_NEIGHBORS
(0)
                   object_dn                : NULL
                   source_dsa_guid          :
00000000-0000-0000-0000-000000000000
rpc fault: DCERPC_FAULT_CANT_PERFORM
ERROR(runtime): DsReplicaGetInfo of type 0 failed - (3221356597, 'The
operation cannot be performed.')
 File "/usr/lib/python2.7/dist-packages/samba/netcmd/drs.py", line
175, in get_neighbours
   self.drsuapi_handle, 1, req1)

==


output from “ samba-tool drs replicate dc2 dc1
OU=Users,OU=Company,DC=my,DC=company,DC=com -d9"

==clipped
.
.
.
    drsuapi_DsReplicaSync: struct drsuapi_DsReplicaSync
       out: struct drsuapi_DsReplicaSync
           result                   : WERR_DS_DRA_BAD_NC
ERROR(<class 'samba.drs_utils.drsException'>): DsReplicaSync
failed - drsException: DsReplicaSync failed (8440, 'WERR_DS_DRA_BAD_NC')
 File "/usr/lib/python2.7/dist-packages/samba/netcmd/drs.py", line
568, in run
   drs_utils.sendDsReplicaSync(server_bind, server_bind_handle, source_dsa_guid,
NC, req_options)
 File "/usr/lib/python2.7/dist-packages/samba/drs_utils.py", line 88,
in sendDsReplicaSync
   raise drsException("DsReplicaSync failed %s" % estr)
ubuntu at dc1:~$ sudo
===

Does anyone have any tips on how to rectify this situation? It’s causing me
great difficulty and stress.


I have a snapshot of 2 of the DC’s before any of these changes were made. Is
there any other option to fix this other than rolling back to the snapshots?

I read *again* on the Wiki that promoting W2012 is not supported and the wiki
says that it will “break replication”. Is that what I’ve done here? How do I fix
replication if it’s been broken by W2k8 R2 or W2012 R2 ?


Have I corrupted the database beyond repair?


If I roll back…, what’s the process for replacing the database on the DC’s that
I didn’t snapshot?

Thanks,

Max
> On Apr 13, 2019, at 1:25 AM, M B via samba <samba at lists.samba.org>
wrote:
> 
> I’m getting errors with replication
> 
> Samba 4.9.3 on Ubuntu 16.04
> 6 Samba DC’s at 3 sites
> 
> I recently joined a Win2k8 R2 server to the domain and promoted it to
domain controller. The replication was working before that, and not working
after.
> 
> I didn’t realize the problems and tried to join a Win2012 R2 server and
promote to DC as well. That also failed with an error about replication
failures.
> 
> 
> sudo samba-tool drs showrepl -d9
> output from “showrepl”
> 
> ==> 
> rpc fault: DCERPC_FAULT_CANT_PERFORM
> ERROR(runtime): DsReplicaGetInfo of type 0 failed - (3221356597, 'The
operation cannot be performed.')
> File "/usr/lib/python2.7/dist-packages/samba/netcmd/drs.py", line
175, in get_neighbours
>  self.drsuapi_handle, 1, req1)
> 
> ==> 
> 
> 
> output from “ samba-tool drs replicate dc2 dc1
OU=Users,OU=Company,DC=my,DC=company,DC=com -d9"
> 
> ==> clipped
> .
> .
> .
>   drsuapi_DsReplicaSync: struct drsuapi_DsReplicaSync
>      out: struct drsuapi_DsReplicaSync
>          result                   : WERR_DS_DRA_BAD_NC
> ERROR(<class 'samba.drs_utils.drsException'>): DsReplicaSync
failed - drsException: DsReplicaSync failed (8440, 'WERR_DS_DRA_BAD_NC')
> File "/usr/lib/python2.7/dist-packages/samba/netcmd/drs.py", line
568, in run
>  drs_utils.sendDsReplicaSync(server_bind, server_bind_handle,
source_dsa_guid, NC, req_options)
> File "/usr/lib/python2.7/dist-packages/samba/drs_utils.py", line
88, in sendDsReplicaSync
>  raise drsException("DsReplicaSync failed %s" % estr)
> ubuntu at dc1:~$ sudo
> ==> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba

>> 
>> I was able to get replication working again by deleting DNS records
that were conflicting in LDAP.
>> 
>> Used ldapcmp tool to find conflicts
>> https://wiki.samba.org/index.php/Samba-tool_ldapcmp
>> 
> 
> Ok, the problem seems to be back again. 
> 
> Ubuntu 16.04, samba 4.9.3
> 
> I’ve checked with: samba-tool ldapcmp 
> between dc1 and every other dc[2-6]
> 
> ldapcmp shows no errors in any combination mentioned above
> 
> Running this command:
> 
> # sudo samba-tool drs showrepl -d9
> 
> ends with this output
> .
> .
> .
> rpc fault: DCERPC_FAULT_CANT_PERFORM
> ERROR(runtime): DsReplicaGetInfo of type 0 failed - (3221356597, 'The
operation cannot be performed.')
> File "/usr/lib/python2.7/dist-packages/samba/netcmd/drs.py", line
175, in get_neighbours
>   self.drsuapi_handle, 1, req1)
> 
> Running this command
> 
> # sudo samba-tool visualize reps
> OR
> # sudo samba-tool visualize ntdsconn
> 
> indicates that replication is occurring, and when I make changes to any
part of the directory, those changes replicate to the other DCs
> 
> Any ideas about what is going on?
> 
> 
> smb.conf
> ===> 
> # Global parameters
> [global]
>       netbios name = DC5
>       realm = MY.COMPANY.COM
>       server role = active directory domain controller
>       server services = s3fs, rpc, nbt, wrepl, ldap, cldap, kdc, drepl,
winbindd, ntp_signd, kcc, dnsupdate
>       workgroup = COMPANY
>       dns zone scavenging = yes
>       idmap_ldb:use rfc2307 = yes
>       tls enabled  = yes
>       tls keyfile  = tls/cert.key
>       tls certfile = tls/cert.pem
>       tls cafile   = tls/int_ca.pem
>       logging = file:2
>       log level = dsdb_audit:4 dsdb_password_audit:4 auth_audit:4 auth:1
passdb:3 winbind:2
> 
> [netlogon]
>       path = /var/lib/samba/sysvol/my.company.com/scripts
>       read only = No
> 
> [sysvol]
>       path = /var/lib/samba/sysvol
>       read only = No
> 
Still having problems here. Does anyone have any ideas about how to address
this?

If I restart samba, I see in the logs after 10-20 seconds, samba is hitting a
PANIC state. Before the panic, I can successfully run

sudo samba-tool drs showrepl

I will get the proper replication status output. 

After the PANIC occurs, then I get the error mentioned above beginning with …
"rpc fault: DCERPC_FAULT_CANT_PERFORM”


The panic looks like this:

[2019/03/31 22:15:29.452501,  0]
../source4/lib/cmdline/popt_common.c:74(popt_s4_talloc_log_fn)
 Bad talloc magic value - unknown value
[2019/03/31 22:15:29.453091,  0] ../lib/util/fault.c:128(smb_panic_default)
 smb_panic_default: PANIC (pid 28449): Bad talloc magic value - unknown value
[2019/03/31 22:15:29.455084,  0] ../lib/util/fault.c:261(log_stack_trace)
 BACKTRACE: 64 stack frames:
  #0 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(log_stack_trace+0x2e)
[0x7f35b40189df]
  #1 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(+0x1f751) [0x7f35b4018751]
  #2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(log_stack_trace+0)
[0x7f35b40189b1]
  #3 /usr/lib/x86_64-linux-gnu/samba/libtalloc.so.2(+0x2825) [0x7f35b3148825]
  #4 /usr/lib/x86_64-linux-gnu/samba/libtalloc.so.2(+0x284b) [0x7f35b314884b]
  #5 /usr/lib/x86_64-linux-gnu/samba/libtalloc.so.2(+0x28c9) [0x7f35b31488c9]
  #6 /usr/lib/x86_64-linux-gnu/samba/libtalloc.so.2(+0x2c6a) [0x7f35b3148c6a]
  #7 /usr/lib/x86_64-linux-gnu/samba/libtalloc.so.2(+0x2e7e) [0x7f35b3148e7e]
  #8 /usr/lib/x86_64-linux-gnu/samba/libtalloc.so.2(+0x5f36) [0x7f35b314bf36]
  #9 /usr/lib/x86_64-linux-gnu/samba/libtalloc.so.2(talloc_strdup+0x49)
[0x7f35b314bfec]
  #10 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(+0x1cd3d) [0x7f35b3dd9d3d]
  #11 /usr/lib/x86_64-linux-gnu/samba/ldb/objectclass_attrs.so(+0x1bc9)
[0x7f359d3dcbc9]
  #12 /usr/lib/x86_64-linux-gnu/samba/ldb/objectclass_attrs.so(+0x22ea)
[0x7f359d3dd2ea]
  #13 /usr/lib/x86_64-linux-gnu/samba/ldb/objectclass_attrs.so(+0x33bc)
[0x7f359d3de3bc]
  #14 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_next_request+0x2fb)
[0x7f35b3dc6820]
  #15 /usr/lib/x86_64-linux-gnu/samba/ldb/instancetype.so(+0x10cf)
[0x7f359ee3d0cf]
  #16 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_next_request+0x2fb)
[0x7f35b3dc6820]
  #17 /usr/lib/x86_64-linux-gnu/samba/ldb/password_hash.so(+0xda85)
[0x7f359c7ada85]
  #18 /usr/lib/x86_64-linux-gnu/samba/ldb/password_hash.so(+0xe0b1)
[0x7f359c7ae0b1]
  #19 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_next_request+0x2fb)
[0x7f35b3dc6820]
  #20 /usr/lib/x86_64-linux-gnu/samba/ldb/samldb.so(+0xcbbc) [0x7f359aab0bbc]
  #21 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_next_request+0x2fb)
[0x7f35b3dc6820]
  #22 /usr/lib/x86_64-linux-gnu/samba/ldb/acl.so(+0x5d04) [0x7f35a0ca6d04]
  #23 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_next_request+0x2fb)
[0x7f35b3dc6820]
  #24 /usr/lib/x86_64-linux-gnu/samba/ldb/descriptor.so(+0x3ddb)
[0x7f35a0284ddb]
  #25 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_next_request+0x2fb)
[0x7f35b3dc6820]
  #26 /usr/lib/x86_64-linux-gnu/samba/ldb/tombstone_reanimate.so(+0x1dcf)
[0x7f35961f1dcf]
  #27 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_next_request+0x2fb)
[0x7f35b3dc6820]
  #28 /usr/lib/x86_64-linux-gnu/samba/ldb/objectclass.so(+0x3a6b)
[0x7f359d5e4a6b]
  #29 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_next_request+0x2fb)
[0x7f35b3dc6820]
  #30 /usr/lib/x86_64-linux-gnu/samba/ldb/audit_log.so(+0x4ca4) [0x7f35a048eca4]
  #31 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_next_request+0x2fb)
[0x7f35b3dc6820]
  #32 /usr/lib/x86_64-linux-gnu/samba/ldb/extended_dn_in.so(+0x22bb)
[0x7f359f85c2bb]
  #33 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_module_done+0x1fc)
[0x7f35b3dc7b6c]
  #34 /usr/lib/x86_64-linux-gnu/samba/ldb/encrypted_secrets.so(+0x3b38)
[0x7f359fa63b38]
  #35 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_module_done+0x1fc)
[0x7f35b3dc7b6c]
  #36 /usr/lib/x86_64-linux-gnu/samba/ldb/operational.so(+0x5fea)
[0x7f359cfd2fea]
  #37 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_module_done+0x1fc)
[0x7f35b3dc7b6c]
  #38 /usr/lib/x86_64-linux-gnu/samba/ldb/extended_dn_out.so(+0x2b1d)
[0x7f359f655b1d]
  #39 /usr/lib/x86_64-linux-gnu/samba/ldb/extended_dn_out.so(+0x3848)
[0x7f359f656848]
  #40 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_module_done+0x1fc)
[0x7f35b3dc7b6c]
  #41
/usr/lib/x86_64-linux-gnu/samba/libdsdb-module-samba4.so(dsdb_next_callback+0x8a)
[0x7f35a44aa68c]
  #42 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_module_done+0x1fc)
[0x7f35b3dc7b6c]
  #43 /usr/lib/x86_64-linux-gnu/samba/ldb/partition.so(+0x3b07) [0x7f359c9b8b07]
  #44 /usr/lib/x86_64-linux-gnu/samba/libldb-key-value-samba4.so(+0x73f8)
[0x7f359e61f3f8]
  #45 /usr/lib/x86_64-linux-gnu/samba/libldb-key-value-samba4.so(+0x7c4b)
[0x7f359e61fc4b]
  #46
/usr/lib/x86_64-linux-gnu/samba/libtevent.so.0(tevent_common_invoke_timer_handler+0x2ae)
[0x7f35b377f3a6]
  #47
/usr/lib/x86_64-linux-gnu/samba/libtevent.so.0(tevent_common_loop_timer_delay+0xc8)
[0x7f35b377f5a9]
  #48 /usr/lib/x86_64-linux-gnu/samba/libtevent.so.0(+0x11a76) [0x7f35b3781a76]
  #49 /usr/lib/x86_64-linux-gnu/samba/libtevent.so.0(+0xe30b) [0x7f35b377e30b]
  #50 /usr/lib/x86_64-linux-gnu/samba/libtevent.so.0(_tevent_loop_once+0x11e)
[0x7f35b3775d95]
  #51 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_wait+0x1c4)
[0x7f35b3de6beb]
  #52 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(+0x299ff) [0x7f35b3de69ff]
  #53 /usr/lib/x86_64-linux-gnu/samba/libldb.so.1(ldb_modify+0xa6)
[0x7f35b3de97b0]
  #54
/usr/lib/x86_64-linux-gnu/samba/libscavenge-dns-records-samba4.so(dns_tombstone_records_zone+0x5ea)
[0x7f35a3c6089b]
  #55
/usr/lib/x86_64-linux-gnu/samba/libscavenge-dns-records-samba4.so(dns_tombstone_records+0x1c6)
[0x7f35a3c60b06]
  #56 /usr/lib/x86_64-linux-gnu/samba/service/kcc.so(+0x6672) [0x7f35a4090672]
  #57 /usr/lib/x86_64-linux-gnu/samba/service/kcc.so(+0x6c42) [0x7f35a4090c42]
  #58 /usr/lib/x86_64-linux-gnu/samba/service/kcc.so(+0x6340) [0x7f35a4090340]
  #59
/usr/lib/x86_64-linux-gnu/samba/libtevent.so.0(tevent_common_invoke_timer_handler+0x2ae)
[0x7f35b377f3a6]
  #60
/usr/lib/x86_64-linux-gnu/samba/libtevent.so.0(tevent_common_loop_timer_delay+0xc8)
[0x7f35b377f5a9]
  #61 /usr/lib/x86_64-linux-gnu/samba/libtevent.so.0(+0x111ef) [0x7f35b37811ef]
  #62 /usr/lib/x86_64-linux-gnu/samba/libtevent.so.0(+0x11af6) [0x7f35b3781af6]
  #63 /usr/lib/x86_64-linux-gnu/samba/libtevent.so.0(+0xe30b) [0x7f35b377e30b]