samba - Nov 2015 - Domain join failure - error during DRS repl ADD: No objectClass found in replPropertyMetaData

On 23/11/15 14:15, Matthew Delfino wrote:
>
> On 2015.11.23, at 3:58 AM, Rowland Penny <rowlandpenny241155 at
gmail.com
> <mailto:rowlandpenny241155 at gmail.com>> wrote:
>
>> On 23/11/15 00:09, Matthew Delfino wrote:
>>>
>>> On 2015.11.22, at 4:43 PM, Matthew Delfino 
>>> <mdelfino.list.samba at KNOCKinc.com 
>>> <mailto:mdelfino.list.samba at KNOCKinc.com> 
>>> <mailto:mdelfino.list.samba at KNOCKinc.com>> wrote:
>>>
>>>>
>>>> On 2015.11.22, at 2:27 PM, Rowland Penny 
>>>> <rowlandpenny241155 at gmail.com
<mailto:rowlandpenny241155 at gmail.com>
>>>> <mailto:rowlandpenny241155 at gmail.com>> wrote:
>>>>
>>>>> On 22/11/15 19:49, Matthew Delfino wrote:
>>>>>> I have 3 domain controllers, running Samba 4.1.6 on
Ubuntu
>>>>>> 14.04.3 LTS in a VMware virtual machine (part of the
package
>>>>>> install available from the "apt-get install
samba" command). My
>>>>>> approach was to do a non-FSMO first, the other non-FSMO
second,
>>>>>> then the FSMO last.
>>>>>>
>>>>>> I started by shutting down all of these three VMs and
doing a
>>>>>> snapshot.
>>>>>>
>>>>>> Next, I needed to backup the files and purge the system
of all
>>>>>> traces of Samba:
>>>>>>
>>>>>> sudo -s
>>>>>> service samba stop
>>>>>> service samba-ad-dc stop
>>>>>> service nmbd stop
>>>>>>
>>>>>> cp -Rp /etc/samba/* 
>>>>>> /root/backup_queue/2015.11.22-SAMBA/root/etc/samba/.
>>>>>> cp -Rp /var/lib/samba/* 
>>>>>>
/root/backup_queue/2015.11.22-SAMBA/root/var/lib/samba/.
>>>>>>
>>>>>> apt-get purge ^samba.*
>>>>>>
>>>>>> Then, I download all the important packages for
compiling from
>>>>>> source on Ubuntu:
>>>>>>
>>>>>> apt-get install acl attr autoconf bison build-essential
debhelper
>>>>>> dnsutils docbook-xml docbook-xsl flex gdb krb5-user
libacl1-dev
>>>>>> libaio-dev libattr1-dev libblkid-dev libbsd-dev
libcap-dev
>>>>>> libgnutls28-dev libjson-perl libldap2-dev
libncurses5-dev
>>>>>> libpam0g-dev libparse-yapp-perl libpopt-dev
libreadline-dev perl
>>>>>> perl-modules pkg-config python-all-dev python-dev 
>>>>>> python-dnspython python-crypto xsltproc zlib1g-dev
>>>>>>
>>>>>> And don’t forget to exit because you don’t want to be
root for
>>>>>> the next commands:
>>>>>>
>>>>>> exit
>>>>>>
>>>>>> Prepare for, download and unpack the source:
>>>>>>
>>>>>> mkdir source
>>>>>> cd source
>>>>>> wget
https://download.samba.org/pub/samba/stable/samba-4.3.1.tar.gz
>>>>>> tar -zxf samba-4.3.1.tar.gz
>>>>>>
>>>>>> Configure and compile (compiles always took, like 15
minutes on
>>>>>> my system):
>>>>>>
>>>>>> cd samba-4.3.1/
>>>>>> ./configure --enable-fhs --prefix=/usr
--sysconfdir=/etc
>>>>>> --localstatedir=/var --enable-debug
>>>>>> make
>>>>>>
>>>>>> Install it:
>>>>>>
>>>>>> sudo -s
>>>>>> make install
>>>>>>
>>>>>> Recover from those backups:
>>>>>>
>>>>>> cp -Rp
/root/backup_queue/2015.11.22-SAMBA/root/etc/samba/*
>>>>>> /etc/samba/.
>>>>>> cp -Rp
/root/backup_queue/2015.11.22-SAMBA/root/var/lib/samba/*
>>>>>> /var/lib/samba/.
>>>>>>
>>>>>> Download the upstart conf and init script from samba
wiki (as per
>>>>>> https://wiki.samba.org/index.php/Samba4/InitScript):
>>>>>>
>>>>>> wget -O /etc/init/samba-ad-dc.conf 
>>>>>>
'http://anonscm.debian.org/gitweb/?p=pkg-samba/samba.git;a=blob_plain;f=debian/samba-ad-dc.upstart;hb=HEAD'
>>>>>>
<http://anonscm.debian.org/gitweb/?p=pkg-samba/samba.git;a=blob_plain;f=debian/samba-ad-dc.upstart;hb=HEAD%27>
>>>>>>
<http://anonscm.debian.org/gitweb/?p=pkg-samba/samba.git;a=blob_plain;f=debian/samba-ad-dc.upstart;hb=HEAD%27>
>>>>>> wget 
>>>>>>
"http://anonscm.debian.org/gitweb/?p=pkg-samba/samba.git;a=blob_plain;f=debian/samba.samba-ad-dc.init;h=3132d2e367675f822342a5b7bc2e50c046aa3b8f;hb=HEAD"
>>>>>> -O /etc/init.d/samba-ad-dc
>>>>>> chmod 755 /etc/init.d/samba-ad-dc
>>>>>> update-rc.d samba-ad-dc defaults
>>>>>>
>>>>>> Restart:
>>>>>>
>>>>>> shutdown -r now
>>>>>>
>>>>>> Log back in at restart. Make sure it’s running:
>>>>>>
>>>>>> service samba-ad-dc status
>>>>>>
>>>>>> In my case, it was running each time (samba-ad-dc
start/running,
>>>>>> process X). Now it’s time to fix stuff:
>>>>>>
>>>>>> sudo samba-tool dbcheck —fix
>>>>>>
>>>>>> It found hundreds of issues and fixed them all. Lastly,
I went on
>>>>>> to check all my work:
>>>>>>
>>>>>> sudo samba-tool drs showrepl
>>>>>>
>>>>>> Says inbound and outbound updates are successful, so
looked good….
>>>>>>
>>>>>> samba-tool ldapcmp ldap://dc00 ldap://dc01
-Uadministrator
>>>>>>
>>>>>> Everything checked out EXCEPT "whenChanged,"
which was off on
>>>>>> scores of records. So, to make myself feel better, I
did this:
>>>>>>
>>>>>> samba-tool ldapcmp --filter="whenChanged"
ldap://dc01 ldap://dc00
>>>>>> -Uadministrator
>>>>>>
>>>>>> I have tested the authentication systems on a few
services that
>>>>>> were bound to these DCs and they all work - so far so
good on
>>>>>> that front.
>>>>>>
>>>>>> I repeated the above steps for the second non-FSMO DC,
then the
>>>>>> FSMO DC. Every one of them is now on Samba 4.3.1.
>>>>>>
>>>>>> But the final test I did, after all the upgrades, was
this:
>>>>>>
>>>>>> sudo samba-tool fsmo show
>>>>>>
>>>>>> ERROR(<type 'exceptions.KeyError'>):
uncaught exception - 'No
>>>>>> such element'
>>>>>> File
"/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py",
>>>>>> line 175, in _run
>>>>>>   return self.run(*args, **kwargs)
>>>>>> File
"/usr/lib/python2.7/dist-packages/samba/netcmd/fsmo.py",
>>>>>> line 395, in run
>>>>>>   domaindnszonesMaster = get_fsmo_roleowner(samdb,
domaindns_dn)
>>>>>> File
"/usr/lib/python2.7/dist-packages/samba/netcmd/fsmo.py",
>>>>>> line 42, in get_fsmo_roleowner
>>>>>>   master_owner = res[0]["fSMORoleOwner"][0]
>>>>>>
>>>>>> Which concerns me. What could I do here to get me out
of this
>>>>>> pickle? Do I have to (re)seize FSMO responsibilities? I
did run
>>>>>> this on the DC that was the FSMO before all these
shenanigans:
>>>>>>
>>>>>> samba-tool fsmo seize --role=all
>>>>>
>>>>> Did you just enter 'samba-tool fsmo seize
--role=all' ?
>>>>> If so, you need to add an adminuser & password i.e. 
>>>>> -UAdministrator and then enter the password when prompted.
>>>>>
>>>>> Rowland
>>>>
>>>> Indeed I did. Hoping you were onto something and this was going
to
>>>> be an easy fix, here’s what I did...
>>>>
>>>> samba-tool fsmo seize --role=all -Uadministrator
>>>>
>>>> This DC already has the 'rid' FSMO role
>>>> This DC already has the 'pdc' FSMO role
>>>> This DC already has the 'naming' FSMO role
>>>> This DC already has the 'infrastructure' FSMO role
>>>> This DC already has the 'schema' FSMO role
>>>> ERROR(<type 'exceptions.KeyError'>): uncaught
exception - 'No such
>>>> element'
>>>> File
"/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py",
>>>> line 175, in _run
>>>>   return self.run(*args, **kwargs)
>>>> File
"/usr/lib/python2.7/dist-packages/samba/netcmd/fsmo.py", line
>>>> 345, in run
>>>>   versionopts, force)
>>>> File
"/usr/lib/python2.7/dist-packages/samba/netcmd/fsmo.py", line
>>>> 301, in seize_dns_role
>>>>   master_owner = get_fsmo_roleowner(samdb, m.dn)
>>>> File
"/usr/lib/python2.7/dist-packages/samba/netcmd/fsmo.py", line
>>>> 42, in get_fsmo_roleowner
>>>>   master_owner = res[0]["fSMORoleOwner"][0]
>>>>
>>
>> You do need the adminuser and password for the dns fsmo roles, but 
>> after revisiting the python code, I don't think this is your
problem.
>> When you try to show or transfer or seize a role, they all run this:
>>
>>    res = samdb.search(roledn,
>>                       scope=ldb.SCOPE_BASE,
attrs=["fSMORoleOwner"])
>>    assert len(res) == 1
>>    master_owner = res[0]["fSMORoleOwner"][0]
>>    return master_owner
>>
>> This is where it seems to be choking for you, but only on the DNS 
>> fsmo roles, can you try running these commands on your DC (as root) , 
>> changing them to match your setup:
>>
>> ldbsearch -H /usr/local/samba/private/sam.ldb -b 
>>
"CN=Infrastructure,DC=DomainDnsZones,DC=samdom,DC=example,DC=com" -s
>> base '(fSMORoleOwner=*)' fSMORoleOwner
>>
>>
>> ldbsearch -H /usr/local/samba/private/sam.ldb -b 
>>
"CN=Infrastructure,DC=ForestDnsZones,DC=samdom,DC=example,DC=com" -s
>> base '(fSMORoleOwner=*)' fSMORoleOwner
>>
>> You should get something like this back for each command:
>>
>> # record 1
>> dn: CN=Infrastructure,DC=DomainDnsZones,DC=samdom,DC=example,DC=com
>> fSMORoleOwner: CN=NTDS 
>> Settings,CN=DC1,CN=Servers,CN=Default-First-Site-Name,C
>> N=Sites,CN=Configuration,DC=samdom,DC=example,DC=com
>>
>> # returned 1 records
>> # 1 entries
>> # 0 referrals
>>
>> Rowland
>
> Thanks for sticking with me on this one, Rowland. Here are the 
> commands, followed by their output, on the DC that should be the FSMO, 
> run as root:
>
> ldbsearch -H /var/lib/samba/private/sam.ldb -b 
> "CN=Infrastructure,DC=DomainDnsZones,DC=mycompany,DC=lan" -s base
> '(fSMORoleOwner=*)' fSMORoleOwner
>
> # record 1
> dn: CN=Infrastructure,DC=DomainDnsZones,DC=mycompany,DC=lan
>
> # returned 1 records
> # 1 entries
> # 0 referrals
>
> ldbsearch -H /var/lib/samba/private/sam.ldb -b 
> "CN=Infrastructure,DC=ForestDnsZones,DC=mycompany,DC=lan" -s base
> '(fSMORoleOwner=*)' fSMORoleOwner
>
> # record 1
> dn: CN=Infrastructure,DC=ForestDnsZones,DC=mycompany,DC=lan
>
> # returned 1 records
> # 1 entries
> # 0 referrals
>
> Matthew
>
>
> ------------------------------------------------------------------------
>
OK, try again, but this time, remove the <fSMORoleOwner> from the end of 
the command, this will dump the entire AD object, I am sure you will 
find that there is no 'fSMORoleOwner' attribute. This is your actual 
problem, why do you not have this FSMO role ?

You have however found a bug in the code, it should print an error 
message if no role owner is found.

Rowland

On 2015.11.23, at 8:32 AM, Rowland Penny <rowlandpenny241155 at gmail.com>
wrote:
> OK, try again, but this time, remove the <fSMORoleOwner> from the end
of the command, this will dump the entire AD object, I am sure you will find
that there is no 'fSMORoleOwner' attribute. This is your actual problem,
why do you not have this FSMO role ?
> 
> You have however found a bug in the code, it should print an error message
if no role owner is found.
> 
> Rowland
Rowland, you nailed it. The 'fSMORoleOwner' attribute was indeed missing
from:

	dn: CN=Infrastructure,DC=DomainDnsZones,DC=mycompany,DC=lan

And

	dn: CN=Infrastructure,DC=ForestDnsZones,DC=mycompany,DC=lan

I used Softerra LDAP Administrator 2015.2 to "Add/Modify Attribute…"
under it’s "Entry" menu while the Infrastructure containers sited
above were selected.

I made sure that the syntax of my entry was correct:

	CN=NTDS
Settings,CN=DC00,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mycompany,DC=lan

And now I get what I expect:

----

	sudo samba-tool fsmo seize --role=all

This DC already has the 'rid' FSMO role
This DC already has the 'pdc' FSMO role
This DC already has the 'naming' FSMO role
This DC already has the 'infrastructure' FSMO role
This DC already has the 'schema' FSMO role
This DC already has the 'domaindns' FSMO role
This DC already has the 'forestdns' FSMO role

----

And this:

----

	sudo ldbsearch -H /var/lib/samba/private/sam.ldb -b
"CN=Infrastructure,DC=DomainDnsZones,DC=knockinc,DC=loc" -s base
'(fSMORoleOwner=*)' fSMORoleOwner

# record 1
dn: CN=Infrastructure,DC=DomainDnsZones,DC=mycompany,DC=lan
fSMORoleOwner: CN=NTDS Settings,CN=DC00,CN=Servers,CN=Default-First-Site-
 Name,CN=Sites,CN=Configuration,DC=mycompany,DC=lan

# returned 1 records
# 1 entries
# 0 referrals

----

Same output for the other one (ForestDnsZones).

----

	sudo ldbsearch -H /var/lib/samba/private/sam.ldb '(invocationId=*)'
--cross-ncs objectguid

# record 1
dn: CN=NTDS
Settings,CN=RHEA,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=knockinc,DC=loc
objectGUID: 6ba7ca4f-291f-4ffe-8403-65fe26a8bfd2

# record 2
dn: CN=NTDS
Settings,CN=ENCELADUS,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=knockinc,DC=loc
objectGUID: 7c40a9c3-be7e-44b5-b2d9-ebe7f97c0517

# record 3
dn: CN=NTDS
Settings,CN=GANYMEDE,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=knockinc,DC=loc
objectGUID: 98b9225b-a5b9-4351-84f8-253762772cc3

# returned 3 records
# 3 entries
# 0 referrals

----

The only other curiosity I have right now is, why are all the
"whenChanged" attributes off between the DCs? Is that normal after a
certain version of Samba, post v4.1.6?

Thanks again, Rowland!

Thanks,
Matthew

©2015 KNOCK, inc.  All rights reserved. KNOCK is a registered trademark of
KNOCK, inc. This message and any attachments contain information, which is
confidential and/or privileged.  If you are not the intended recipient, please
refrain from any disclosure, copying, distribution or use of this information.
 Please be aware that such actions are prohibited.  If you have received this
transmission in error, kindly notify the sender by e-mail.  Your cooperation is
appreciated.

On 23/11/15 15:51, Matthew Delfino wrote:
>
> On 2015.11.23, at 8:32 AM, Rowland Penny <rowlandpenny241155 at
gmail.com
> <mailto:rowlandpenny241155 at gmail.com>> wrote:
>> OK, try again, but this time, remove the <fSMORoleOwner> from the
end
>> of the command, this will dump the entire AD object, I am sure you 
>> will find that there is no 'fSMORoleOwner' attribute. This is
your
>> actual problem, why do you not have this FSMO role ?
>>
>> You have however found a bug in the code, it should print an error 
>> message if no role owner is found.
>>
>> Rowland
>
> Rowland, you nailed it. The 'fSMORoleOwner' attribute was indeed 
> missing from:
>
> dn: CN=Infrastructure,DC=DomainDnsZones,DC=mycompany,DC=lan
>
> And
>
> dn: CN=Infrastructure,DC=ForestDnsZones,DC=mycompany,DC=lan
>
> I used Softerra LDAP Administrator 2015.2 to "Add/Modify
Attribute…"
> under it’s "Entry" menu while the Infrastructure containers sited
> above were selected.
>
> I made sure that the syntax of my entry was correct:
>
> CN=NTDS 
>
Settings,CN=DC00,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=mycompany,DC=lan
>
> And now I get what I expect:
>
> ----
>
> sudo samba-tool fsmo seize --role=all
>
> This DC already has the 'rid' FSMO role
> This DC already has the 'pdc' FSMO role
> This DC already has the 'naming' FSMO role
> This DC already has the 'infrastructure' FSMO role
> This DC already has the 'schema' FSMO role
> This DC already has the 'domaindns' FSMO role
> This DC already has the 'forestdns' FSMO role
>
> ----
>
> And this:
>
> ----
>
> sudo ldbsearch -H /var/lib/samba/private/sam.ldb -b 
> "CN=Infrastructure,DC=DomainDnsZones,DC=knockinc,DC=loc" -s base 
> '(fSMORoleOwner=*)' fSMORoleOwner
>
> # record 1
> dn: CN=Infrastructure,DC=DomainDnsZones,DC=mycompany,DC=lan
> fSMORoleOwner: CN=NTDS Settings,CN=DC00,CN=Servers,CN=Default-First-Site-
>  Name,CN=Sites,CN=Configuration,DC=mycompany,DC=lan
>
> # returned 1 records
> # 1 entries
> # 0 referrals
>
> ----
>
> Same output for the other one (ForestDnsZones).
>
> ----
>
> sudo ldbsearch -H /var/lib/samba/private/sam.ldb '(invocationId=*)'
> --cross-ncs objectguid
>
> # record 1
> dn: CN=NTDS 
>
Settings,CN=RHEA,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=knockinc,DC=loc
> objectGUID: 6ba7ca4f-291f-4ffe-8403-65fe26a8bfd2
>
> # record 2
> dn: CN=NTDS 
>
Settings,CN=ENCELADUS,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=knockinc,DC=loc
> objectGUID: 7c40a9c3-be7e-44b5-b2d9-ebe7f97c0517
>
> # record 3
> dn: CN=NTDS 
>
Settings,CN=GANYMEDE,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=knockinc,DC=loc
> objectGUID: 98b9225b-a5b9-4351-84f8-253762772cc3
>
> # returned 3 records
> # 3 entries
> # 0 referrals
>
> ----
>
> The only other curiosity I have right now is, why are all the 
> "whenChanged" attributes off between the DCs? Is that normal
after a
> certain version of Samba, post v4.1.6?
>
>
I can help you with that as well, find ldapcmp.py on your system, open 
it in your favourite editor, find this line:

                 # "whenChanged", # This is implicitly replicated

Make it look like this:

                 "whenChanged",

Save & close the file.

You won't get the error again, whoever changed it last time seems to 
have made a mistake, if you look here:

https://msdn.microsoft.com/en-us/library/windows/desktop/ms680921%28v=vs.85%29.aspx

You will see that 'whenChanged' is not replicated.

Rowland