samba - Dec 2016 - AD, 4.5.0, DRS or deletion question

Hi all,

I have a strange behaviour on our AD.

DC=ForestDnsZones,DC=ad,DC=domain,DC=tld
        Authentification\DC208 via RPC
                DSA object GUID: 20f711ed-cb02-4543-badb-28d3ed4c4ae1
                Last attempt @ NTTIME(0) was successful
                0 consecutive failure(s).
                Last success @ NTTIME(0)

DC=ForestDnsZones,DC=ad,DC=domain,DC=tld
        NTDS DN: CN=NTDS
Settings\0ADEL:9a4b7c54-ae49-484b-baaa-524621ddb52e,CN=DC208\0ADEL:107ed4a0-9197-469c-a3e0-d981d9e266b6,CN=Servers,CN=Authentification,CN=Sites,CN=Configuration,DC=ad,DC=infra,DC=dgfip
                DSA object GUID: 9a4b7c54-ae49-484b-baaa-524621ddb52e
                Last attempt @ Tue Dec 13 17:38:52 2016 CET failed, result
2 (WERR_BADFILE)
                367327 consecutive failure(s).
                Last success @ Fri Nov 18 10:58:08 2016 CET

This is an extract of "samba-tool drs showrepl" on one server. One
block
lists update against a real DC (DC208) and the other one shows update
against deleted DC, the same DC208 but its old object.

This deleted object appears on every DIT except "schema" and
"configuration" (both haven't changed for a while).

Most important question: How can this be possible?

A lower important question: How to get rid of that? Except modifying
tombstoneLifetime.

Cheers : )

No answer from anyone from the community so I managed by myself, answering
also questions by myself.

So...
Question 1: How can a DC relies on deleted object to perform replication?
That is a bug from Samba (the new KCC?).
Sorry to say that but what else? Deleted object are objects which are not
in use. They have something to do perhaps with replication but they MUST
NOT be used as valid source (or destination) for replication. So, a bug.


Question 2: as previously said I don't want to have to modify the
tombstoneLifetime because this implies modifying the schema which is not
something to perform regularly.
What if:
- object deletion can't be performed
- removal of this replication path is not possible because the path does
not really exist (not listed into KCC CONNECTION OBJECTS section of drs
showrepl, not existing into "AD sites and services" MSC)
- forcing replication between DC does not solve the issue
?

To solve this issue I stopped Samba service on DC having the issue and then
I copied manually (using a simple "scp") the DIT files from one
working DC
to this broken DC.
After restarting the Samba service this DC has no issue.

I know this kind of copy is not advised by Samba team, but else can we do
in such case?

And why is it not advised as it works?
The main argument against this kind of copy was "DB are not exactly the
same".
If we can trust "samba-tool ldapcmp" the only difference between same
DIT
(by DIT I mean DOMAINDNSZONES, FORESTDNSZONES files in private/sam.ldb.d/
directory) on different DCs is the "whenChanged" attribute. And this
attribute is not replicated because it does not really matter.
So the difference between databases on different DCs does not really matter
and copy can be performed.

After having shared here what solution what applied I'll go back to work
and start writing that down in our docs.

If you read that all, thank you for your interest : )

mathias

2016-12-13 18:05 GMT+01:00 mathias dufresne <infractory at gmail.com>:
> Hi all,
>
> I have a strange behaviour on our AD.
>
> DC=ForestDnsZones,DC=ad,DC=domain,DC=tld
>         Authentification\DC208 via RPC
>                 DSA object GUID: 20f711ed-cb02-4543-badb-28d3ed4c4ae1
>                 Last attempt @ NTTIME(0) was successful
>                 0 consecutive failure(s).
>                 Last success @ NTTIME(0)
>
> DC=ForestDnsZones,DC=ad,DC=domain,DC=tld
>         NTDS DN: CN=NTDS Settings\0ADEL:9a4b7c54-ae49-
> 484b-baaa-524621ddb52e,CN=DC208\0ADEL:107ed4a0-9197-
> 469c-a3e0-d981d9e266b6,CN=Servers,CN=Authentification,
> CN=Sites,CN=Configuration,DC=ad,DC=infra,DC=dgfip
>                 DSA object GUID: 9a4b7c54-ae49-484b-baaa-524621ddb52e
>                 Last attempt @ Tue Dec 13 17:38:52 2016 CET failed, result
> 2 (WERR_BADFILE)
>                 367327 consecutive failure(s).
>                 Last success @ Fri Nov 18 10:58:08 2016 CET
>
> This is an extract of "samba-tool drs showrepl" on one server.
One block
> lists update against a real DC (DC208) and the other one shows update
> against deleted DC, the same DC208 but its old object.
>
> This deleted object appears on every DIT except "schema" and
> "configuration" (both haven't changed for a while).
>
> Most important question: How can this be possible?
>
> A lower important question: How to get rid of that? Except modifying
> tombstoneLifetime.
>
> Cheers : )
>
>

On Thu, 2016-12-15 at 14:25 +0100, mathias dufresne via samba
wrote:
> No answer from anyone from the community so I managed by myself,
> answering
> also questions by myself.
> 
> So...
> Question 1: How can a DC relies on deleted object to perform
> replication?
> That is a bug from Samba (the new KCC?).
> Sorry to say that but what else? Deleted object are objects which are
> not
> in use. They have something to do perhaps with replication but they
> MUST
> NOT be used as valid source (or destination) for replication. So, a
> bug.
It just means that it has in the past replicated from this DC, as the
repsFrom entry is by GUID.  It is transformed into a DN at presentation
time for display.
> 
> Question 2: as previously said I don't want to have to modify the
> tombstoneLifetime because this implies modifying the schema which is
> not
> something to perform regularly.
> What if:
> - object deletion can't be performed
> - removal of this replication path is not possible because the path
> does
> not really exist (not listed into KCC CONNECTION OBJECTS section of
> drs
> showrepl, not existing into "AD sites and services" MSC)
> - forcing replication between DC does not solve the issue
> ?
> 
> To solve this issue I stopped Samba service on DC having the issue
> and then
> I copied manually (using a simple "scp") the DIT files from one
> working DC
> to this broken DC.
> After restarting the Samba service this DC has no issue.
You now have a CORRUPT database, will experience EXTREME pain in the
near future as you try and untangle this mess.  NEVER copy sam.ldb
files between different hosts, as it contains non-replicated metadata. 

Please shut down the "DC having the issue" at once, and remove it from
the domain using 'samba-tool domain demote --remove-other-dead-server=' 
from your other working DC.

To be clear, I use such strong words to ensure that others do not
follow in your footprints.

The only circumstances in which a sam.ldb file should be copied between
hosts is when simply doing a life-and-shift move of the whole Samba
installation onto a new host with the same name.  You can not copy the
DB between replicas. 

Sorry,

Andrew Bartlett