samba - Jan 2017 - Initial replication halts with "The handle is invalid." (msDS-NC-Replica-Locations corrupted?)

Quoting Adam Tauno Williams via samba <samba at
lists.samba.org>:
> Quoting Rowland Penny via samba <samba at lists.samba.org>:
>>> samba-tool's dbcheck finds only two errors in cn=Configuration,
but it
>>> does not repair them.  These appear to be references to an
original,
>>> long since demoted, DC.  But these values appear in neither the
>>> results of an ldbsearch or via ADSI edit.
>> Have a look at this bug report:
>> https://bugzilla.samba.org/show_bug.cgi?id=12385
> Thanks.  I have checked out master and built it [**1];  Bug#12297  
> [**2] indicates I can run samba-tool from the build of the checkout.  
>  I assume tombstone expunge, even for CN=Configuration, runs against  
> sam.ldb???  So the correct syntax is -
> [ADC28 samba]# bin/samba-tool domain tombstones expunge -H  
> /var/lib/samba/private/sam.ldb

This does not appear to have reaped any links.


[root at larkin28 samba]# bin/samba-tool domain tombstones expunge -d 10  
-H /var/lib/samba/private/sam.ldb
...
lpcfg_servicenumber: couldn't find ldb
Initial schema load needed, as we have no existing schema, seq_num: 64
schema_fsmo_init: we are master[no] updates allowed[no]
Initial schema load needed, as we have no existing schema, seq_num: 64
schema_fsmo_init: we are master[no] updates allowed[no]
gendb_search_v: CN=Directory Service,CN=Windows  
NT,CN=Services,CN=Configuration,DC=micore,DC=us  
objectClass=nTDSService -> 1
Doing a full scan on CN=Configuration,DC=micore,DC=us and looking for  
deleted objects
Doing a full scan on DC=micore,DC=us and looking for deleted objects
Doing a full scan on DC=DomainDnsZones,DC=micore,DC=us and looking for  
deleted objects
Doing a full scan on DC=ForestDnsZones,DC=micore,DC=us and looking for  
deleted objects
Removed 0 objects and 0 links successfully

On Mon, 16 Jan 2017 14:03:37 -0500
Adam Tauno Williams <awilliam at whitemice.org> wrote:
> Quoting Adam Tauno Williams via samba <samba at lists.samba.org>:
> > Quoting Rowland Penny via samba <samba at lists.samba.org>:
> >>> samba-tool's dbcheck finds only two errors in
cn=Configuration,
> >>> but it does not repair them.  These appear to be references to
an
> >>> original, long since demoted, DC.  But these values appear in
> >>> neither the results of an ldbsearch or via ADSI edit.
> >> Have a look at this bug report:
> >> https://bugzilla.samba.org/show_bug.cgi?id=12385
> > Thanks.  I have checked out master and built it [**1];  Bug#12297  
> > [**2] indicates I can run samba-tool from the build of the
> > checkout. I assume tombstone expunge, even for CN=Configuration,
> > runs against sam.ldb???  So the correct syntax is -
> > [ADC28 samba]# bin/samba-tool domain tombstones expunge -H  
> > /var/lib/samba/private/sam.ldb
> 
> 
> This does not appear to have reaped any links.
> 
> 
> [root at larkin28 samba]# bin/samba-tool domain tombstones expunge -d
> 10 -H /var/lib/samba/private/sam.ldb
> ...
> lpcfg_servicenumber: couldn't find ldb
> Initial schema load needed, as we have no existing schema, seq_num: 64
> schema_fsmo_init: we are master[no] updates allowed[no]
> Initial schema load needed, as we have no existing schema, seq_num: 64
> schema_fsmo_init: we are master[no] updates allowed[no]
> gendb_search_v: CN=Directory Service,CN=Windows  
> NT,CN=Services,CN=Configuration,DC=micore,DC=us  
> objectClass=nTDSService -> 1
> Doing a full scan on CN=Configuration,DC=micore,DC=us and looking
> for deleted objects
> Doing a full scan on DC=micore,DC=us and looking for deleted objects
> Doing a full scan on DC=DomainDnsZones,DC=micore,DC=us and looking
> for deleted objects
> Doing a full scan on DC=ForestDnsZones,DC=micore,DC=us and looking
> for deleted objects
> Removed 0 objects and 0 links successfully
> 
> 
I 'think' you are meant to patch Samba, then totally recompile it
i.e. ./configure && make && make install, then run samba-tool.

From the way I read it, you are supposed to ignore bug 12297

Rowland

Quoting Rowland Penny via samba <samba at
lists.samba.org>:
> I 'think' you are meant to patch Samba, then totally recompile it
> i.e. ./configure && make && make install, then run
samba-tool.
I did that

   git clone https://git.samba.org/samba.git
   cd samba/
   yum install gcc libacl-devel python-devel gnutls-devel openldap-devel
   ./configure
   make
   ls bin/samba-tool

The patch mentioned in Bug#12385  
<https://bugzilla.samba.org/show_bug.cgi?id=12385> appears to already  
be applied to master;  attempting to "patch -p1 < patchfile"
results
in an already-applied warning.

I ran the tool as -

   bin/samba-tool domain tombstones expunge -d 10 -H  
/var/lib/samba/private/sam.ldb
> From the way I read it, you are supposed to ignore bug 12297
I only referred to that bug as Mr. Bartlett there said """This
patch
is to code used by both, but the test I was after just needs you to  
build and run 'bin/samba-tool domain tombstones expunge' from an  
in-tree build of master, after applying this.
No install will be required for this step."""

On Mon, 2017-01-16 at 14:03 -0500, Adam Tauno Williams via samba
wrote:
> Quoting Adam Tauno Williams via samba <samba at lists.samba.org>:
> > Quoting Rowland Penny via samba <samba at lists.samba.org>:
> > > > samba-tool's dbcheck finds only two errors in
cn=Configuration,
> > > > but it
> > > > does not repair them.  These appear to be references to an
> > > > original,
> > > > long since demoted, DC.  But these values appear in neither
the
> > > > results of an ldbsearch or via ADSI edit.
> > > 
> > > Have a look at this bug report:
> > > https://bugzilla.samba.org/show_bug.cgi?id=12385
> > 
> > Thanks.  I have checked out master and built it [**1];  Bug#12297  
> > [**2] indicates I can run samba-tool from the build of the
> > checkout.  
> >  I assume tombstone expunge, even for CN=Configuration, runs
> > against  
> > sam.ldb???  So the correct syntax is -
> > [ADC28 samba]# bin/samba-tool domain tombstones expunge -H  
> > /var/lib/samba/private/sam.ldb
> 
> 
> This does not appear to have reaped any links.
> 
> 
> [root at larkin28 samba]# bin/samba-tool domain tombstones expunge -d
> 10  
> -H /var/lib/samba/private/sam.ldb
> ...
> lpcfg_servicenumber: couldn't find ldb
> Initial schema load needed, as we have no existing schema, seq_num:
> 64
> schema_fsmo_init: we are master[no] updates allowed[no]
> Initial schema load needed, as we have no existing schema, seq_num:
> 64
> schema_fsmo_init: we are master[no] updates allowed[no]
> gendb_search_v: CN=Directory Service,CN=Windows  
> NT,CN=Services,CN=Configuration,DC=micore,DC=us  
> objectClass=nTDSService -> 1
> Doing a full scan on CN=Configuration,DC=micore,DC=us and looking
> for  
> deleted objects
> Doing a full scan on DC=micore,DC=us and looking for deleted objects
> Doing a full scan on DC=DomainDnsZones,DC=micore,DC=us and looking
> for  
> deleted objects
> Doing a full scan on DC=ForestDnsZones,DC=micore,DC=us and looking
> for  
> deleted objects
> Removed 0 objects and 0 links successfully
Thanks.  When you run 
bin/samba-tool dbcheck -H /var/lib/samba/private/sam.ldb --cross-ncs
is it less scary now?  We tried to remove some of the scare factor for
this one, it is 'mostly harmless'.

In your earlier mail you said the output was:

[root at larkin27 ~]# samba-tool dbcheck --cross-ncs --fix
--yes
> Checking 8411 objects
> ERROR: no target object found for GUID component for msDS-NC-Replica-
> Locations in object CN=3ad6381a-9725-4e28-8157-
> a5a3fde68a43,CN=Partitions,CN=Configuration,DC=micore,DC=us -
>
<GUID=7d3f95a5cdfa1246b1fb2fcd16e5f877>;<RMD_ADDTIME=1300000651000000
> 00
> >
;<RMD_CHANGETIME=130000065100000000>;<RMD_FLAGS=0>;<RMD_INVOCID=01d
> > b57> 
>
fd8d4ddd469aee9cbd36abb3e1>;<RMD_LOCAL_USN=5160>;<RMD_ORIGINATING_USN
> =3
> 630>;<RMD_VERSION=0>;CN=NTDS
> Settings,CN=BARBEL,CN=Servers,CN=Default-
> First-Site-Name,CN=Sites,CN=Configuration,DC=micore,DC=us
> Not removing dangling forward link
> ERROR: no target object found for GUID component for msDS-NC-Replica-
> Locations in object CN=55b4d7f1-b1b1-4843-ae00-
> 7908adf44ffa,CN=Partitions,CN=Configuration,DC=micore,DC=us -
>
<GUID=7d3f95a5cdfa1246b1fb2fcd16e5f877>;<RMD_ADDTIME=1300000651000000
> 00
> >
;<RMD_CHANGETIME=130000065100000000>;<RMD_FLAGS=0>;<RMD_INVOCID=01d
> > b57> 
>
fd8d4ddd469aee9cbd36abb3e1>;<RMD_LOCAL_USN=5134>;<RMD_ORIGINATING_USN
> =3
> 629>;<RMD_VERSION=0>;CN=NTDS
> Settings,CN=BARBEL,CN=Servers,CN=Default-
> First-Site-Name,CN=Sites,CN=Configuration,DC=micore,DC=us
> Not removing dangling forward link
So, what is happening here is that we have a non-deleted link to an
object that does not exist.  We have so far been very conservative in
removing such links, because logic does exist to remove them when the
target object is deleted.  We are concerned they may point at something
that is yet to be replicated (eg if you replicated with --critical-
objects-only).  They may point outside our current replicated partition
(once we get subdomains), and in general they are not something we are
quite as clear as being totally redundant.  (ie, we don't want to
automatically delete possible information)

The tombstones expunge command above is related to a slightly different
issue, which is that in the past, when we deleted an object, we deleted
(marked with RMD_FLAGS=1, rather than just removed) links to the
deleted object (which it self was marked with isDeleted=TRUE and had
\0DEL:GUID put in the name).

Time would pass and in 180 days the target object would be deleted, but
until 4.5.2 the link wouldn't be removed.  Also with 4.5.0, we started
checking more of these links, not just ignoring them.  So the reason
the tombstones expunge was suggested is that IF it was RMD_FLAGS=1, and
deleted at the same time as the target, the expunge would also find it
180 days old, and tidy the link away... 

Now, the fact that you have replication failures suggests this is a
very real problem for you.  It may or may not actually be this object,
but clearly you want to try.  I'll try and get you a command to remove
the links with the 'vanish links' control, so you can test it out on a
backup.  

Thanks,,

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

>> > sam.ldb???  So the correct syntax is -
>> > [ADC28 samba]# bin/samba-tool domain tombstones expunge -H  
>> > /var/lib/samba/private/sam.ldb
>> This does not appear to have reaped any links.
>> [root at larkin28 samba]# bin/samba-tool domain tombstones expunge -d
>> 10  
>> -H /var/lib/samba/private/sam.ldb
>> ...
>> lpcfg_servicenumber: couldn't find ldb
>> Initial schema load needed, as we have no existing schema, seq_num:
>> 64
>> schema_fsmo_init: we are master[no] updates allowed[no]
>> Initial schema load needed, as we have no existing schema, seq_num:
>> 64
>> schema_fsmo_init: we are master[no] updates allowed[no]
>> gendb_search_v: CN=Directory Service,CN=Windows  
>> NT,CN=Services,CN=Configuration,DC=micore,DC=us  
>> objectClass=nTDSService -> 1
>> Doing a full scan on CN=Configuration,DC=micore,DC=us and looking
>> for  
>> deleted objects
>> Doing a full scan on DC=micore,DC=us and looking for deleted objects
>> Doing a full scan on DC=DomainDnsZones,DC=micore,DC=us and looking
>> for  
>> deleted objects
>> Doing a full scan on DC=ForestDnsZones,DC=micore,DC=us and looking
>> for  
>> deleted objects
>> Removed 0 objects and 0 links successfully
> Thanks.  When you run 
> bin/samba-tool dbcheck -H /var/lib/samba/private/sam.ldb --cross-ncs
> is it less scary now?  We tried to remove some of the scare factor for
> this one, it is 'mostly harmless'.
No, it still contains the same errors concerning msDS-NC-Replica-Locations
> Now, the fact that you have replication failures suggests this is a
> very real problem for you.
Yes; fairly urgently.

Would any additional information be helpful?  I can attempt again to  
promote the 2008R2 DC and get a packet capture from between the servers.
> It may or may not actually be this object, but clearly you want to  
> try.  I'll try and get you a command to remove the links with the  
> 'vanish links' control, so you can test it out on a backup.