samba - Mar 2019 - Replication and KCC problems on upgrade

Hello all-

I am trying to upgrade a old domain to a newer version. The old DCs are a custom
compiled version of Samba, so instead of upgrading the DCs in place, the plan is
to upgrade by joining new DCs to the domain, replicating data and then shutting
down the old ones after transferring the FSMO roles.

I had the new DC (dc3, version 4.9.4-12) replicating to the other DCs (dc0,
versions 4.0.6-12 and dc1 and dc2, version 4.0.6-8) with no known issues.
Specifically, "samba-tool dbcheck --cross-ncs" reported no issues on
any DCs, "samba-tool drs showrepl" reported no issues on any DCs and
"samba-tool ldapcmp" returned without errors on dc0 compared to all
other DCs.

Clients and all functions seemed to be behaving appropriately.

At that point, I demoted dc1 and dc2. The demote command did not return errors.

However, now dc0 and dc3 are having issues. Specifically, "samba-tool
ldapcmp" run on dc0 compared to dc3 returns:

Comparing:
'CN=6a8bca7c-3069-4ada-be59-100c970d59fd,CN=NTDS
Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=example,DC=com'
[dc0]
'CN=6a8bca7c-3069-4ada-be59-100c970d59fd,CN=NTDS
Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=example,DC=com'
[dc3]
    Attributes found only in dc3:
        fromServer
    FAILED

This error is NOT shown when comparing dc3 to dc0.

While poking around at this, I also found that "samba-tool drs kcc
dc3" (run on dc0) returns no errors, but "samba-tool drs kcc dc0"
(run on dc3) fails with:

Failed to bind to uuid e3514235-4b06-11d1-ab04-00c04fc2dcd2 for
ncacn_ip_tcp:::1[49152,seal,target_hostname=dc0,abstract_syntax=e3514235-4b06-11d1-ab04-00c04fc2dcd2/0x00000004,localaddress=::1]
NT_STATUS_UNSUCCESSFUL
ERROR(<call 'samba.drs_utils.drsException'>): DRS connection to
dc0 failed - drsException: DRS connection to dc0 failed: (3221225473,
'{Operation Failed} The requested operation was unsuccessful.')


Anyone have more information one why the errors are one-sided and what I can do
about this?


Thanks,

Mike Ray

On Thu, 28 Feb 2019 18:04:50 -0600 (CST)
Mike Ray via samba <samba at lists.samba.org> wrote:
> Hello all-
> 
> I am trying to upgrade a old domain to a newer version. The old DCs
> are a custom compiled version of Samba, so instead of upgrading the
> DCs in place, the plan is to upgrade by joining new DCs to the
> domain, replicating data and then shutting down the old ones after
> transferring the FSMO roles.
> 
> I had the new DC (dc3, version 4.9.4-12) replicating to the other DCs
> (dc0, versions 4.0.6-12 and dc1 and dc2, version 4.0.6-8) with no
> known issues. Specifically, "samba-tool dbcheck --cross-ncs"
reported
> no issues on any DCs, "samba-tool drs showrepl" reported no
issues on
> any DCs and "samba-tool ldapcmp" returned without errors on dc0
> compared to all other DCs.
> 
> Clients and all functions seemed to be behaving appropriately.
> 
> At that point, I demoted dc1 and dc2. The demote command did not
> return errors.
> 
> However, now dc0 and dc3 are having issues. Specifically, "samba-tool
> ldapcmp" run on dc0 compared to dc3 returns:
> 
> Comparing:
> 'CN=6a8bca7c-3069-4ada-be59-100c970d59fd,CN=NTDS
>
Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=example,DC=com'
[dc0]
> 'CN=6a8bca7c-3069-4ada-be59-100c970d59fd,CN=NTDS
>
Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=example,DC=com'
[dc3]
> Attributes found only in dc3: fromServer FAILED
> 
> This error is NOT shown when comparing dc3 to dc0.
> 
> While poking around at this, I also found that "samba-tool drs kcc
> dc3" (run on dc0) returns no errors, but "samba-tool drs kcc
> dc0" (run on dc3) fails with:
> 
> Failed to bind to uuid e3514235-4b06-11d1-ab04-00c04fc2dcd2 for
>
ncacn_ip_tcp:::1[49152,seal,target_hostname=dc0,abstract_syntax=e3514235-4b06-11d1-ab04-00c04fc2dcd2/0x00000004,localaddress=::1]
> NT_STATUS_UNSUCCESSFUL ERROR(<call
'samba.drs_utils.drsException'>):
> DRS connection to dc0 failed - drsException: DRS connection to dc0
> failed: (3221225473, '{Operation Failed} The requested operation was
> unsuccessful.')
> 
> 
> Anyone have more information one why the errors are one-sided and
> what I can do about this?
> 
> 
> Thanks,
> 
> Mike Ray
> 
I wonder if this has anything to do with the 'you cannot upgrade
directly from 4.7.x to 4.9.x' bug ? 
I know this might seem strange, but try running ldbedit on your new DC.

Rowland

----- On Mar 1, 2019, at 3:35 AM, samba samba at lists.samba.org
wrote:
> 
> I wonder if this has anything to do with the 'you cannot upgrade
> directly from 4.7.x to 4.9.x' bug ?

I was not aware of this bug. Do you think I should scrap this upgrade and try
again jumping like so? 4.0.6-12 -> 4.7 -> 4.8 -> 4.9

> I know this might seem strange, but try running ldbedit on your new DC.

"ldbedit -H ldap://dc3 -UAdministrator" seemed to run without issue
and let me modify an entry.

> 
> Rowland
> 
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba

After running the ldbedit command, I checked the state of the DCs.

"samba-tool dbcheck --cross-ncs" returned nothing on dc0; on dc3 it
returned:

Checking 6916 objects
NOTE: old (due to rename or delete) DN string component for fromServer in object
CN=6a8bca7c-3069-4ada-be59-100c970d59fd,CN=NTDS
Settings,CN=DC3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=x-es,DC=com
- CN=NTDS
Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=x-es,DC=com
Not fixing old string component
NOTE: old (due to rename or delete) DN string component for lastKnownParent in
object
CN=ee835988-3702-420f-a935-d12d8f977f47\0ADEL:adc1836d-adba-4785-8cd7-73065c3e6d53,CN=Deleted
Objects,CN=Configuration,DC=x-es,DC=com - CN=NTDS
Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=x-es,DC=com
Not fixing old string component
NOTE: old (due to rename or delete) DN string component for lastKnownParent in
object
CN=ab5dcd50-9fd9-4db7-bc59-e4f9b55fcbd7\0ADEL:0f50abd8-b289-412e-9ae6-4299bbe06d66,CN=Deleted
Objects,CN=Configuration,DC=x-es,DC=com - CN=NTDS
Settings,CN=DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=x-es,DC=com
Not fixing old string component
NOTE: old (due to rename or delete) DN string component for lastKnownParent in
object
CN=9ea6a27c-ae95-4fac-a00f-33ea2c2a9dab\0ADEL:bff63288-ef7b-4b1a-8cad-74f4c88db301,CN=Deleted
Objects,CN=Configuration,DC=x-es,DC=com - CN=NTDS
Settings,CN=DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=x-es,DC=com
Not fixing old string component
NOTE: old (due to rename or delete) DN string component for lastKnownParent in
object
CN=5d421d22-2216-4475-beb2-8cc46a514cb9\0ADEL:323679f7-d893-451e-ab10-3d8e08e05843,CN=Deleted
Objects,CN=Configuration,DC=x-es,DC=com - CN=NTDS
Settings,CN=DC2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=x-es,DC=com
Not fixing old string component
NOTE: old (due to rename or delete) DN string component for lastKnownParent in
object
CN=2d38127c-7f95-42f7-aaf2-a42f86d54aab\0ADEL:27e13ab1-9930-4363-9d56-2704f275eed3,CN=Deleted
Objects,CN=Configuration,DC=x-es,DC=com - CN=NTDS
Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=x-es,DC=com
Not fixing old string component
NOTE: old (due to rename or delete) DN string component for lastKnownParent in
object
CN=53c549bb-6964-4bbe-bd24-33f40c9ef5f3\0ADEL:1bc38396-2162-47d9-8780-29177548e208,CN=Deleted
Objects,CN=Configuration,DC=x-es,DC=com - CN=NTDS
Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=x-es,DC=com
Not fixing old string component
Checked 6916 objects (0 errors)

Running "samba-tool dbcheck --cross-ncs --fix" removed these notes
without issue and they did not show up on a subsequent run.

The "fromServer" object note is interesting as that was the attribute
(and CN) listed as a difference in the ldapcmp.

However, running "samba-tool ldapcmp dc0 dc3 configuration
--filter=msDS-NcType,serverState,subrefs" still errors on the fromServer
attribute.

Running "samba-tool drs kcc dc0" on dc3 still breaks with the DRS
connection failure.