samba - Oct 2017 - samba 4.7.0 replication errors

Hallo,
we have 5 ADDCs. All of them did run with sernet-samba 4.6.7.
I updated 4 of them to sernet-samba 4.7.0, one after the other, checked
replication, everything seemed to be ok.
One day later a colleague wanted to delete a lot of users with a
powershell-script and since then the
replication doesnt work anymore. (Im sure the script is not the problem, but it
seemes like it triggered something)

All samba-servers with version 4.7.0 report errors with at least one other ADDC
like

DC=domain,DC=de
 Default-First-Site-Name\ISAMBA4-2 via RPC
   DSA object GUID: 5dc32731-e914-486d-96f1-ce065ff956bf
   Last attempt @ Fri Sep 29 10:37:24 2017 CEST failed, result 58
(WERR_BAD_NET_RESP)
   358 consecutive failure(s).
   Last success @ Thu Sep 28 10:18:16 2017 CEST


The command "samba-tool dbcheck --cross-ncs --fix --yes" reports
hundreds of errors like

   ERROR: orphaned backlink attribute 'memberOf' ...

The dbcheck-command says, it fixed the problems, but when I execute again, a lot
of the same error comes again ( I can not say, if the same entries are
effected).

The log.samba has a lot of entries like
   [2017/09/29 10:26:15.502219,  0]
../source4/dsdb/repl/drepl_out_helpers.c:959(dreplsrv_op_pull_source_apply_changes_trigger)
       Failed to commit objects:
WERR_GEN_FAILURE/NT_STATUS_INVALID_NETWORK_RESPONSE


If I make the dbcheck on the last server with version 4.6.7, this errors dont
appear.

How do I get the replication to work again ?

Is the error "orphaned backlink attribute" the reason, why replication
doesnt work anymore ?
And if so, do I have to fix all groups manually like said in a similar problem
from the post "Samba 4.7.0 replication issue: failed get spanning tree
edges" ?
(https://lists.samba.org/archive/samba/2017-September/211225.html)

Can you provide a bit more logs? At first glance, it doesn't seem quite 
related to group memberships.


Cheers,

Garming

On 29/09/17 22:07, gizmo via samba wrote:
> Hallo,
> we have 5 ADDCs. All of them did run with sernet-samba 4.6.7.
> I updated 4 of them to sernet-samba 4.7.0, one after the other, checked
replication, everything seemed to be ok.
> One day later a colleague wanted to delete a lot of users with a
powershell-script and since then the
> replication doesnt work anymore. (Im sure the script is not the problem,
but it seemes like it triggered something)
>
> All samba-servers with version 4.7.0 report errors with at least one other
ADDC like
>
> DC=domain,DC=de
>   Default-First-Site-Name\ISAMBA4-2 via RPC
>     DSA object GUID: 5dc32731-e914-486d-96f1-ce065ff956bf
>     Last attempt @ Fri Sep 29 10:37:24 2017 CEST failed, result 58
(WERR_BAD_NET_RESP)
>     358 consecutive failure(s).
>     Last success @ Thu Sep 28 10:18:16 2017 CEST
>
>
> The command "samba-tool dbcheck --cross-ncs --fix --yes" reports
hundreds of errors like
>
>     ERROR: orphaned backlink attribute 'memberOf' ...
>
> The dbcheck-command says, it fixed the problems, but when I execute again,
a lot of the same error comes again ( I can not say, if the same entries are
effected).
>
> The log.samba has a lot of entries like
>     [2017/09/29 10:26:15.502219,  0]
../source4/dsdb/repl/drepl_out_helpers.c:959(dreplsrv_op_pull_source_apply_changes_trigger)
>         Failed to commit objects:
WERR_GEN_FAILURE/NT_STATUS_INVALID_NETWORK_RESPONSE
>
>
> If I make the dbcheck on the last server with version 4.6.7, this errors
dont appear.
>
> How do I get the replication to work again ?
>
> Is the error "orphaned backlink attribute" the reason, why
replication doesnt work anymore ?
> And if so, do I have to fix all groups manually like said in a similar
problem from the post "Samba 4.7.0 replication issue: failed get spanning
tree edges" ?
> (https://lists.samba.org/archive/samba/2017-September/211225.html)
>

On Mon, 2 Oct 2017 09:59:47 +1300
Garming Sam via samba <samba at lists.samba.org> wrote:
> Can you provide a bit more logs? At first glance, it doesn't seem
> quite related to group memberships.
> 
> 
> Cheers,
> 
> Garming
> 
> On 29/09/17 22:07, gizmo via samba wrote:
> > Hallo,
> > we have 5 ADDCs. All of them did run with sernet-samba 4.6.7.
> > I updated 4 of them to sernet-samba 4.7.0, one after the other,
> > checked replication, everything seemed to be ok. One day later a
> > colleague wanted to delete a lot of users with a powershell-script
> > and since then the replication doesnt work anymore. (Im sure the
> > script is not the problem, but it seemes like it triggered
> > something)
> >
> > All samba-servers with version 4.7.0 report errors with at least
> > one other ADDC like
> >
> > DC=domain,DC=de
> >   Default-First-Site-Name\ISAMBA4-2 via RPC
> >     DSA object GUID: 5dc32731-e914-486d-96f1-ce065ff956bf
> >     Last attempt @ Fri Sep 29 10:37:24 2017 CEST failed, result 58
> > (WERR_BAD_NET_RESP) 358 consecutive failure(s).
> >     Last success @ Thu Sep 28 10:18:16 2017 CEST
> >
> >
> > The command "samba-tool dbcheck --cross-ncs --fix --yes"
reports
> > hundreds of errors like
> >
> >     ERROR: orphaned backlink attribute 'memberOf' ...
> >
> > The dbcheck-command says, it fixed the problems, but when I execute
> > again, a lot of the same error comes again ( I can not say, if the
> > same entries are effected).
> >
> > The log.samba has a lot of entries like
> >     [2017/09/29 10:26:15.502219,
> > 0]
../source4/dsdb/repl/drepl_out_helpers.c:959(dreplsrv_op_pull_source_apply_changes_trigger)
> > Failed to commit objects:
> > WERR_GEN_FAILURE/NT_STATUS_INVALID_NETWORK_RESPONSE
> >
> >
> > If I make the dbcheck on the last server with version 4.6.7, this
> > errors dont appear.
> >
> > How do I get the replication to work again ?
> >
> > Is the error "orphaned backlink attribute" the reason, why
> > replication doesnt work anymore ? And if so, do I have to fix all
> > groups manually like said in a similar problem from the post
"Samba
> > 4.7.0 replication issue: failed get spanning tree edges" ?
> > (https://lists.samba.org/archive/samba/2017-September/211225.html)
> >
> 
> 
Aren’t the orphaned backlinks an artefact of a previous fix ? I seem to
remember they have always been there, but the 'fix' just exposed them.

It might help if we could see the powershell script, just how were the
users deleted, the other question is: why was a powershell script used,
when it would have been easier to write a bash script around
'samba-tool user delete'

Rowland

On Mon, 2017-10-02 at 09:59 +1300, Garming Sam via samba
wrote:
> Can you provide a bit more logs? At first glance, it doesn't seem quite
> related to group memberships.
> 
I agree, we need more logs here.  Turn up the log level and see what
the error causing that final error is.  

However, take care not to publish confidential details like staff names
and sensitive attributes like unicodePwd or supplimentalCredentials to
a public mailing list. 

Running 'samba-tool drs clone-dc-database' against one of the DCs would
be very instructive.  This does the same thing as a fresh join, but
without adding any DC objects. 

The dbcheck errors you mention are interesting.  Backlinks are only
implicitly transferred over DRS replication, but if they are very wrong
perhaps the update of them failed.  What did the powershell script do? 
Did it just delete users, or did it try to remove them from the group
first?

If replication broke only after user/group modification, then this may
be due to a latent DB issue, not detected after the initial upgrade
because nothing read or modified those DB entries.  Once they were
touched the issue became 'live'.

In particular, Samba 4.7.0 includes code to sort links like member
within an attribute.  The process to modify the group list after the
upgrade to sorted links might fail if the DB wasn't clean.

A downgrade to Samba 4.6 should be safe in the meantime, we haven't
changed the DB format and it is much less strict in this area (the
change was made to improve performance), however we would really like
to understand the issue more.

Thanks!

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

Hallo,

I encountered a similar problem.

I created a test environment with two domain controllers (copy from a 
working environment). I tried to join a read-only domain controller. 
Unsuccessfully. Samba-tool fell with a error:

....

added interface ens192 ip=192.168.59.5 bcast=192.168.59.255 
netmask=255.255.255.0
resolve_lmhosts: Attempting lmhosts lookup for name dcg2.unn.global<0x20>
Starting GENSEC mechanism spnego
Starting GENSEC submechanism gssapi_krb5
Received smb_krb5 packet of length 277
Received smb_krb5 packet of length 162
gensec_gssapi: NO credentials were delegated
GSSAPI Connection will be cryptographically signed
Adding DNS A record RODCG3.unn.global for IPv4 IP: 192.168.59.5
ERROR(ldb): uncaught exception - connection to remote LDAP server dropped?
   File 
"/usr/local/samba/lib64/python2.7/site-packages/samba/netcmd/__init__.py",
line 176, in _run
     return self.run(*args, **kwargs)
   File 
"/usr/local/samba/lib64/python2.7/site-packages/samba/netcmd/domain.py",
line 667, in run
     dns_backend=dns_backend)
   File
"/usr/local/samba/lib64/python2.7/site-packages/samba/join.py",
line 1444, in join_RODC
     ctx.do_join()
   File
"/usr/local/samba/lib64/python2.7/site-packages/samba/join.py",
line 1394, in do_join
     ctx.cleanup_old_join()
   File
"/usr/local/samba/lib64/python2.7/site-packages/samba/join.py",
line 270, in cleanup_old_join
     ctx.cleanup_old_accounts(force=force)
   File
"/usr/local/samba/lib64/python2.7/site-packages/samba/join.py",
line 216, in cleanup_old_accounts
     attrs=["msDS-krbTgtLink", "objectSID"])
Deleted CN=RODCG3,OU=Domain Controllers,DC=unn,DC=global
Deleted CN=RODC Connection (FRS),CN=NTDS 
Settings,CN=RODCG3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=unn,DC=global
Deleted CN=NTDS 
Settings,CN=RODCG3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=unn,DC=global
Deleted 
CN=RODCG3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=unn,DC=global
Adding CN=RODCG3,OU=Domain Controllers,DC=unn,DC=global
Adding CN=krbtgt_RODCG3,CN=Users,DC=unn,DC=global
Got krbtgt_name=krbtgt_62809
Renaming CN=krbtgt_RODCG3,CN=Users,DC=unn,DC=global to 
CN=krbtgt_62809,CN=Users,DC=unn,DC=global
Adding 
CN=RODCG3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=unn,DC=global
Adding CN=NTDS 
Settings,CN=RODCG3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=unn,DC=global
Adding CN=RODC Connection (FRS),CN=NTDS 
Settings,CN=RODCG3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=unn,DC=global
Adding SPNs to CN=RODCG3,OU=Domain Controllers,DC=unn,DC=global
Setting account password for RODCG3$
Enabling account
Calling bare provision
Provision OK for domain DN DC=unn,DC=global
Starting replication
Replicating critical objects from the base DN of the domain
Done with always replicated NC (base, config, schema)
Replicating DC=DomainDnsZones,DC=unn,DC=global
Replicating DC=ForestDnsZones,DC=unn,DC=global
Committing SAM database
Join failed - cleaning up

After that, replication stopped working.


When executing the samba-tool dbcheck --cross-ncs, appear messages that 
orphaned backlinks was corrected, but replication not working.

showrepl from dcg1:
Default-First-Site-Name\DCG1
DSA Options: 0x00000001
DSA object GUID: 3c1a24b4-8e75-408f-9724-e047d15d0c5c
DSA invocationId: 8c8dbb4e-901a-4261-85c7-cd15ab6b0acd

==== INBOUND NEIGHBORS ===
DC=ForestDnsZones,DC=unn,DC=global
     Default-First-Site-Name\DCG2 via RPC
         DSA object GUID: ac2074ab-0d12-44d0-ab0b-ad172ff2c131
         Last attempt @ Mon Oct 23 13:32:12 2017 MSK was successful
         0 consecutive failure(s).
         Last success @ Mon Oct 23 13:32:12 2017 MSK

CN=Schema,CN=Configuration,DC=unn,DC=global
     Default-First-Site-Name\DCG2 via RPC
         DSA object GUID: ac2074ab-0d12-44d0-ab0b-ad172ff2c131
         Last attempt @ Mon Oct 23 13:32:12 2017 MSK was successful
         0 consecutive failure(s).
         Last success @ Mon Oct 23 13:32:12 2017 MSK

DC=unn,DC=global
     Default-First-Site-Name\DCG2 via RPC
         DSA object GUID: ac2074ab-0d12-44d0-ab0b-ad172ff2c131
         Last attempt @ Mon Oct 23 13:32:15 2017 MSK failed, result 58 
(WERR_BAD_NET_RESP)
         1991 consecutive failure(s).
         Last success @ Mon Oct 16 14:59:54 2017 MSK

CN=Configuration,DC=unn,DC=global
     Default-First-Site-Name\DCG2 via RPC
         DSA object GUID: ac2074ab-0d12-44d0-ab0b-ad172ff2c131
         Last attempt @ Mon Oct 23 13:32:14 2017 MSK was successful
         0 consecutive failure(s).
         Last success @ Mon Oct 23 13:32:14 2017 MSK

DC=DomainDnsZones,DC=unn,DC=global
     Default-First-Site-Name\DCG2 via RPC
         DSA object GUID: ac2074ab-0d12-44d0-ab0b-ad172ff2c131
         Last attempt @ Mon Oct 23 13:32:26 2017 MSK was successful
         0 consecutive failure(s).
         Last success @ Mon Oct 23 13:32:26 2017 MSK

==== OUTBOUND NEIGHBORS ===
DC=ForestDnsZones,DC=unn,DC=global
     Default-First-Site-Name\DCG2 via RPC
         DSA object GUID: ac2074ab-0d12-44d0-ab0b-ad172ff2c131
         Last attempt @ NTTIME(0) was successful
         0 consecutive failure(s).
         Last success @ NTTIME(0)

CN=Schema,CN=Configuration,DC=unn,DC=global
     Default-First-Site-Name\DCG2 via RPC
         DSA object GUID: ac2074ab-0d12-44d0-ab0b-ad172ff2c131
         Last attempt @ NTTIME(0) was successful
         0 consecutive failure(s).
         Last success @ NTTIME(0)

DC=unn,DC=global
     Default-First-Site-Name\DCG2 via RPC
         DSA object GUID: ac2074ab-0d12-44d0-ab0b-ad172ff2c131
         Last attempt @ NTTIME(0) was successful
         0 consecutive failure(s).
         Last success @ NTTIME(0)

CN=Configuration,DC=unn,DC=global
     Default-First-Site-Name\DCG2 via RPC
         DSA object GUID: ac2074ab-0d12-44d0-ab0b-ad172ff2c131
         Last attempt @ NTTIME(0) was successful
         0 consecutive failure(s).
         Last success @ NTTIME(0)

DC=DomainDnsZones,DC=unn,DC=global
     Default-First-Site-Name\DCG2 via RPC
         DSA object GUID: ac2074ab-0d12-44d0-ab0b-ad172ff2c131
         Last attempt @ NTTIME(0) was successful
         0 consecutive failure(s).
         Last success @ NTTIME(0)

==== KCC CONNECTION OBJECTS ===
Connection --
     Connection name: 06431000-9a51-4959-b9db-714d477c7655
     Enabled        : TRUE
     Server DNS name : dcg2.unn.global
     Server DN name  : CN=NTDS 
Settings,CN=DCG2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=unn,DC=global
         TransportType: RPC
         options: 0x00000001
Warning: No NC replicated for Connection!

showrepl from dcg2:

Default-First-Site-Name\DCG2
DSA Options: 0x00000001
DSA object GUID: ac2074ab-0d12-44d0-ab0b-ad172ff2c131
DSA invocationId: 3d430322-787a-4a7b-9bfc-686112e28394

==== INBOUND NEIGHBORS ===
DC=ForestDnsZones,DC=unn,DC=global
     Default-First-Site-Name\DCG1 via RPC
         DSA object GUID: 3c1a24b4-8e75-408f-9724-e047d15d0c5c
         Last attempt @ Mon Oct 23 13:35:02 2017 MSK was successful
         0 consecutive failure(s).
         Last success @ Mon Oct 23 13:35:02 2017 MSK

CN=Schema,CN=Configuration,DC=unn,DC=global
     Default-First-Site-Name\DCG1 via RPC
         DSA object GUID: 3c1a24b4-8e75-408f-9724-e047d15d0c5c
         Last attempt @ Mon Oct 23 13:35:02 2017 MSK was successful
         0 consecutive failure(s).
         Last success @ Mon Oct 23 13:35:02 2017 MSK

DC=unn,DC=global
     Default-First-Site-Name\DCG1 via RPC
         DSA object GUID: 3c1a24b4-8e75-408f-9724-e047d15d0c5c
         Last attempt @ Mon Oct 23 13:35:03 2017 MSK was successful
         0 consecutive failure(s).
         Last success @ Mon Oct 23 13:35:03 2017 MSK

CN=Configuration,DC=unn,DC=global
     Default-First-Site-Name\DCG1 via RPC
         DSA object GUID: 3c1a24b4-8e75-408f-9724-e047d15d0c5c
         Last attempt @ Mon Oct 23 13:35:03 2017 MSK was successful
         0 consecutive failure(s).
         Last success @ Mon Oct 23 13:35:03 2017 MSK

DC=DomainDnsZones,DC=unn,DC=global
     Default-First-Site-Name\DCG1 via RPC
         DSA object GUID: 3c1a24b4-8e75-408f-9724-e047d15d0c5c
         Last attempt @ Mon Oct 23 13:35:02 2017 MSK was successful
         0 consecutive failure(s).
         Last success @ Mon Oct 23 13:35:02 2017 MSK

==== OUTBOUND NEIGHBORS ===
DC=ForestDnsZones,DC=unn,DC=global
     Default-First-Site-Name\DCG1 via RPC
         DSA object GUID: 3c1a24b4-8e75-408f-9724-e047d15d0c5c
         Last attempt @ NTTIME(0) was successful
         0 consecutive failure(s).
         Last success @ NTTIME(0)

CN=Schema,CN=Configuration,DC=unn,DC=global
     Default-First-Site-Name\DCG1 via RPC
         DSA object GUID: 3c1a24b4-8e75-408f-9724-e047d15d0c5c
         Last attempt @ NTTIME(0) was successful
         0 consecutive failure(s).
         Last success @ NTTIME(0)

DC=unn,DC=global
     Default-First-Site-Name\DCG1 via RPC
         DSA object GUID: 3c1a24b4-8e75-408f-9724-e047d15d0c5c
         Last attempt @ Mon Oct 23 13:33:04 2017 MSK was successful
         0 consecutive failure(s).
         Last success @ Mon Oct 23 13:33:04 2017 MSK

CN=Configuration,DC=unn,DC=global
     Default-First-Site-Name\DCG1 via RPC
         DSA object GUID: 3c1a24b4-8e75-408f-9724-e047d15d0c5c
         Last attempt @ NTTIME(0) was successful
         0 consecutive failure(s).
         Last success @ NTTIME(0)

DC=DomainDnsZones,DC=unn,DC=global
     Default-First-Site-Name\DCG1 via RPC
         DSA object GUID: 3c1a24b4-8e75-408f-9724-e047d15d0c5c
         Last attempt @ NTTIME(0) was successful
         0 consecutive failure(s).
         Last success @ NTTIME(0)

==== KCC CONNECTION OBJECTS ===
Connection --
     Connection name: 160a8775-1734-4931-bcb6-213310952226
     Enabled        : TRUE
     Server DNS name : dcg1.unn.global
     Server DN name  : CN=NTDS 
Settings,CN=DCG1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=unn,DC=global
         TransportType: RPC
         options: 0x00000001
Warning: No NC replicated for Connection!

both my DC are the same: dcg2 (FSMO)
centos 7.3.1611 x64
samba 4.7.0 compiled  ./configure --exec-prefix=/usr --sysconfdir=/etc 
--libdir=/usr/lib64 --localstatedir=/var --enable-fhs 
--with-lockdir=/var/cache/samba --with-modulesdir=/usr/lib64/samba
DNS: SAMBA_INTERNAL
smb.conf:
# Global parameters
[global]
     netbios name = DCG2
     realm = UNN.GLOBAL
     workgroup = UNN
     server role = active directory domain controller
     ldap server require strong auth = no
         dns forwarder = xx.xx.xx.xx, xx.xx.xx.xx
         idmap_ldb:use rfc2307 = yes
         log level = 3 auth:5 winbind:5 passdb:5
#       passdb:5 auth:5
         host msdfs = yes
         tls enabled  = yes
         tls keyfile  = tls/dcg2Key2.pem
         tls certfile = tls/dcg2Cert2.pem
         tls cafile   = tls/luca_root.pem
     ntlm auth = yes

[netlogon]
     path = /var/lib/samba/sysvol/unn.global/scripts
     read only = No

[sysvol]
     path = /var/lib/samba/sysvol
     read only = No

smb.conf for rodcg3 is auto-generated I just add tls options (this does 
not affect the joining)

There are 64000 objects in my AD.

What do you advise? I would like to build a working environment :).



01.10.2017 23:59, Garming Sam via samba пишет:
> Can you provide a bit more logs? At first glance, it doesn't seem 
> quite related to group memberships.
>
>
> Cheers,
>
> Garming
>
> On 29/09/17 22:07, gizmo via samba wrote:
>> Hallo,
>> we have 5 ADDCs. All of them did run with sernet-samba 4.6.7.
>> I updated 4 of them to sernet-samba 4.7.0, one after the other, 
>> checked replication, everything seemed to be ok.
>> One day later a colleague wanted to delete a lot of users with a 
>> powershell-script and since then the
>> replication doesnt work anymore. (Im sure the script is not the 
>> problem, but it seemes like it triggered something)
>>
>> All samba-servers with version 4.7.0 report errors with at least one 
>> other ADDC like
>>
>> DC=domain,DC=de
>>   Default-First-Site-Name\ISAMBA4-2 via RPC
>>     DSA object GUID: 5dc32731-e914-486d-96f1-ce065ff956bf
>>     Last attempt @ Fri Sep 29 10:37:24 2017 CEST failed, result 58 
>> (WERR_BAD_NET_RESP)
>>     358 consecutive failure(s).
>>     Last success @ Thu Sep 28 10:18:16 2017 CEST
>>
>>
>> The command "samba-tool dbcheck --cross-ncs --fix --yes"
reports
>> hundreds of errors like
>>
>>     ERROR: orphaned backlink attribute 'memberOf' ...
>>
>> The dbcheck-command says, it fixed the problems, but when I execute 
>> again, a lot of the same error comes again ( I can not say, if the 
>> same entries are effected).
>>
>> The log.samba has a lot of entries like
>>     [2017/09/29 10:26:15.502219,  0] 
>>
../source4/dsdb/repl/drepl_out_helpers.c:959(dreplsrv_op_pull_source_apply_changes_trigger)
>>         Failed to commit objects: 
>> WERR_GEN_FAILURE/NT_STATUS_INVALID_NETWORK_RESPONSE
>>
>>
>> If I make the dbcheck on the last server with version 4.6.7, this 
>> errors dont appear.
>>
>> How do I get the replication to work again ?
>>
>> Is the error "orphaned backlink attribute" the reason, why 
>> replication doesnt work anymore ?
>> And if so, do I have to fix all groups manually like said in a 
>> similar problem from the post "Samba 4.7.0 replication issue:
failed
>> get spanning tree edges" ?
>> (https://lists.samba.org/archive/samba/2017-September/211225.html)
>>
>
>
-- 
Evgeniy