samba - Sep 2017 - Server GC/name.dom/dom is not registered with our KDC: Miscellaneous failure (see text): Server (GC/name/dom@DOM) unknown

Hai Sven,  
> -----Oorspronkelijk bericht-----
> Van: Sven Schwedas [mailto:sven.schwedas at tao.at] 
> Verzonden: dinsdag 5 september 2017 17:13
> Aan: L.P.H. van Belle
> CC: samba at lists.samba.org
> Onderwerp: Re: [Samba] Server GC/name.dom/dom is not 
> registered with our KDC: Miscellaneous failure (see text): 
> Server (GC/name/dom at DOM) unknown
> 
> On 2017-09-05 16:52, L.P.H. van Belle wrote:
> > Yes, if you flexible with reinstalling, you could..
> 
> I don't want another quick and dirty solution that turns out 
> to break half a year down the line, I'm fine with nuking half 
> my DCs if that means getting to a clean state.
No,  i dont want a quick and dirty solution for you. You need to get a good fix.
> 
> Besides, recreating containers is faster than manually 
> messing around in /var/lib on each one of them.
> 
> > I suggest the following, move fsmo roles to villach-dc and 
> check database replications.
> 
> DB replication is already spewing errors, what am I to look out for?
Ok, get my check db script, run it from any dc. And post me the output. 
https://github.com/thctlo/samba4/blob/master/samba-check-db-repl.sh 
With the output, we should be able to see which servers have the best replicated
database.
( the script uses the standaard samba tools, but all in one go) 
> 
> > Remove the most faulty one first, graz-dc-1b, from the 
> domain. ( check 
> > and cleanup DNS and AD! Very important )
> 
> What to check for? What to clean up?
Ah, thats hard to tell, this depends a bit on the errors.
I search/look for the left overs, first with RSAT tools and samba-tool, then
with ApacheStudio.
I look for hostnames/UUID/ things like that, but this is only done if all other
options did not work.
But it depends on the errors/warnings i see/get. 
> 
> > You dont have to reinstall the complete os, just cleanup as 
> told, and reprovisioning that server again. 
> 
> Adding a new DC with the same hostname as the old DC is what 
> got me into trouble last time. I'll pass up on that offer.
Ok, but i know the correct steps to do this, its all in the correct order and
when to remove where/what.
I can save you the time to reinstall the OS, you can re-use the os, just dont
reuse the same hostname.
But, if its not an option for you anymore, thats ok, that what you want. 
> 
> >>> Then remove a failty server and re-add it as a new installed
DC.
> >>> ( the good DS with FSMO)
> >>> First backup: /var/lib/samba/private/secrets.keytab
> >>> Remove the incorrect entries from keytab file with ktutil rkt 
> >>> /var/lib/samba/private/secrets.keytab
> >>> list -e -t
> >>
> >> Might as well just nuke graz-dc-sem and add a complete new DC from
> >> scratch, no?
> > No, and yes, but i preffer no, not needed (yet). 
> > Start with the keytab cleanup
> > Check the dns record if the uuid A PTR and hostnames 
> resolve to the correct server. 
> > If thats the case, then no, cleanup of keytab is, i think, 
> sufficient. 
> 
> Just to confirm the order: Clean up the keytab, if that 
> doesn't work, start removing servers?
Almost. Backup then ...   Cleanup keytab of the server. 
> 
> > Yes, if its really a mess. ;-)
> > Then, first a an new DC, then remove, just make sure you 
> always have 2 
> > dc's up and running (correctly)
> 
> Servers in Villach seem to run fine, thank $DEITY, so I'll 
> leave them alone for now.
Ok, thats good, run the check-db script and post the complete output for me. 
> 
> >>> Now re-provision and you should have correct working DC's
again.
> >>>
> >>> ! Before re-provisioning, make sure all OLD records dns and
> >> AD are gone. 
> >>
> >> I still have undeleteable replication records from the last time I
> >> had to nuke a DC, nobody replied to my emails on that issue.
> > 
> > Ok, now, im out of office in about 10 min, but mail that 
> subject for me again> I'll have a look.
> 
> First message on that topic:
> https://lists.samba.org/archive/samba/2017-March/207429.html
Ok, this one, track down both uuid's, checkout which which hostname belongs
with these.
Basicly https://wiki.samba.org/index.php/Demoting_a_Samba_AD_DC 
In based on the Demoteing wiki example. 
I do the same steps as shown there. Now the last three pictures on the site
shows where to look.
At the site and Service, go through very folder, and check if it is as it should
be.

https://lists.samba.org/archive/samba/2017-March/207460.html
Between 2 and 3, there the problem starts
>>  – I noticed a typo in the server's `netbios name` setting,
corrected it, and restarted the DC.
3. Yes, for the new hostname, the old hostname is as left over in the ADDC DB
and/or DNS.

This name GRAZ-DC-BIS, and the name with the typo. The GUID for these is where
to look info.

Step 6. ah the point of origin of you problems with of the current post? 

7. dnsmasq? Ok, i just hope these are not running on the DC's. 
> Last message, where I mentioned the replication bug:
> https://lists.samba.org/archive/samba/2017-April/207918.html
> 
> > Own and if you dont use it, ApacheDirectoryStudio can help 
> a lot with cleanup of these kind of things. 
> 
> Currently I'm using the ADSI MMC snap-in, any downsides 
> compared to ADS?
I dont know, never used ADS :-/ 

Track done these : GUID's, the hostname's, ipnumbers A and PTR records 
7e4973ba-4093-4523-a70f-7caa4845e34d 
d613fa11-064b-4bcc-ab01-20264f870f47
(how, see Verifying and Creating the objectGUID Record 
https://wiki.samba.org/index.php/Verifying_and_Creating_a_DC_DNS_Record)
And as suggested, do this first through the RSAT tools or samba-tools, try
removing them with these tools first.
(how https://wiki.samba.org/index.php/Demoting_a_Samba_AD_DC) 
Open the Active Directory Users and Computers application and set it to advanced
view.

Now, i think the last left overs can be removed with : Active Directory Sites
and Services and DNS manager.
And now check them all, every folder, take your time. 

Then when its done, i run samba-tool dbcheck again per
server.
> 
> > But just make sure you know what you delete, for you mess 
> up the AD even more. 
> 
> That why I'm not touching anything without a full list.
Yes, good, im pro that, the more info we get the better we can help you. 


Greetz, 

Louis

On 2017-09-06 09:28, L.P.H. van Belle wrote:
>>> I suggest the following, move fsmo roles to villach-dc and 
>> check database replications.
>>
>> DB replication is already spewing errors, what am I to look out for?
> Ok, get my check db script, run it from any dc. And post me the output. 
Output attached. Seems like all servers but graz-dc-sem have some stuff
wrong, just wonderful.
> https://github.com/thctlo/samba4/blob/master/samba-check-db-repl.sh 
Just FYI, something's wonky with the script. Bash trips up on CRLF line
endings (everywhere) and 0xC2 control characters (ll. 260-261) sprinkled
through the file. Had to clean that up before it actually ran without
syntax errors.
>>> Remove the most faulty one first, graz-dc-1b, from the 
>> domain. ( check 
>>> and cleanup DNS and AD! Very important )
>>
>> What to check for? What to clean up?
> Ah, thats hard to tell, this depends a bit on the errors.
> I search/look for the left overs, first with RSAT tools and samba-tool,
then with ApacheStudio.
> I look for hostnames/UUID/ things like that, but this is only done if all
other options did not work.
> But it depends on the errors/warnings i see/get. 
None of the differences look like they could cause the replication
errors, looks more like they're just the results.
>>>>> Then remove a failty server and re-add it as a new
installed DC.
>>>>> ( the good DS with FSMO)
>>>>> First backup: /var/lib/samba/private/secrets.keytab
>>>>> Remove the incorrect entries from keytab file with ktutil
rkt
>>>>> /var/lib/samba/private/secrets.keytab
>>>>> list -e -t
>>>>
>>>> Might as well just nuke graz-dc-sem and add a complete new DC
from
>>>> scratch, no?
>>> No, and yes, but i preffer no, not needed (yet). 
>>> Start with the keytab cleanup
>>> Check the dns record if the uuid A PTR and hostnames 
>> resolve to the correct server. 
>>> If thats the case, then no, cleanup of keytab is, i think, 
>> sufficient. 
>>
>> Just to confirm the order: Clean up the keytab, if that 
>> doesn't work, start removing servers?
> Almost. Backup then ...   Cleanup keytab of the server. 
I'll try that next, unless you find something shady in the report.
>>> Yes, if its really a mess. ;-)
>>> Then, first a an new DC, then remove, just make sure you 
>> always have 2 
>>> dc's up and running (correctly)
>>
>> Servers in Villach seem to run fine, thank $DEITY, so I'll 
>> leave them alone for now.
> Ok, thats good, run the check-db script and post the complete output for
me.
> 
>>
>>>>> Now re-provision and you should have correct working
DC's again.
>>>>>
>>>>> ! Before re-provisioning, make sure all OLD records dns and
>>>> AD are gone. 
>>>>
>>>> I still have undeleteable replication records from the last
time I
>>>> had to nuke a DC, nobody replied to my emails on that issue.
>>>
>>> Ok, now, im out of office in about 10 min, but mail that 
>> subject for me again> I'll have a look.
>>
>> First message on that topic:
>> https://lists.samba.org/archive/samba/2017-March/207429.html
> Ok, this one, track down both uuid's, checkout which which hostname
belongs with these.
Going by the output of `samba-tool drs showrepl` and sam.ldb, the GUIDs
are as follows:

graz-dc-sem	160f5a53-5c29-4a83-aeee-6cb1dbabeed7
graz-dc-bis	7e4973ba-4093-4523-a70f-7caa4845e34d (not in sam.ldb)
graz-dc-1b	bcffbad8-1add-46b9-bf69-90e52c0f09ea
villach-dc-sem	eb5f9772-cd8f-4cde-9629-f1898c94aedd
villach-dc-bis	e1569c90-50f9-4bb5-bd85-79145e3ff6fd

villach-sem/bis want to replicate to the dead and removed graz-dc-bis
for some reason, but don't have its GUID in their sam.ldb either.
> Basicly https://wiki.samba.org/index.php/Demoting_a_Samba_AD_DC 
> In based on the Demoteing wiki example. 
> I do the same steps as shown there. Now the last three pictures on the site
shows where to look.
> At the site and Service, go through very folder, and check if it is as it
should be.
I already checked it back then, and double checked it now. ADUC, ADSS
and DNS are all three clean and contain no reference to either the old
server name or its GUID.
> https://lists.samba.org/archive/samba/2017-March/207460.html
> Between 2 and 3, there the problem starts
>>>  – I noticed a typo in the server's `netbios name` setting,
corrected it, and restarted the DC.
> 3. Yes, for the new hostname, the old hostname is as left over in the ADDC
DB and/or DNS.
> 
> This name GRAZ-DC-BIS, and the name with the typo. The GUID for these is
where to look info.
Yeah, but where?
> Step 6. ah the point of origin of you problems with of the current post? 
Hell if I know, but might be.
> 7. dnsmasq? Ok, i just hope these are not running on the DC's. 
It's running on the gateways to cache the DCs' responses and handle some
auxiliary stuff. ad.tao.at and all subdomains are transparently
forwarded to the DCs, no difference in responses for _msdsc etc. between
them and querying the DCs directly.
>> Last message, where I mentioned the replication bug:
>> https://lists.samba.org/archive/samba/2017-April/207918.html
>>
>>> Own and if you dont use it, ApacheDirectoryStudio can help 
>> a lot with cleanup of these kind of things. 
>>
>> Currently I'm using the ADSI MMC snap-in, any downsides 
>> compared to ADS?
> I dont know, never used ADS :-/ 
> 
> Track done these : GUID's, the hostname's, ipnumbers A and PTR
records
> 7e4973ba-4093-4523-a70f-7caa4845e34d 
> d613fa11-064b-4bcc-ab01-20264f870f47
> (how, see Verifying and Creating the objectGUID Record 
https://wiki.samba.org/index.php/Verifying_and_Creating_a_DC_DNS_Record)
Neither of the two return any results on any DC (or on the dnsmasq
servers), but the GUIDs for the 4 currently existing DCs work fine. This
matches the data from the RSAT tools.

-- 
Mit freundlichen Grüßen, / Best Regards,
Sven Schwedas, Systemadministrator
Mail/XMPP sven.schwedas at tao.at | Skype sven.schwedas
TAO Digital | Lendplatz 45 | A8020 Graz
https://www.tao-digital.at | Tel +43 680 301 7167
-------------- next part --------------
No password for user AD\Administrator was set in this script!
Please enter the password for AD\Administrator :
Running with with console output
Checking the DC_With_FSMO (graz-dc-sem) with SAMBA DC: villach-dc-sem.ad.tao.at
villach-dc-bis.ad.tao.at
graz-dc-1b.ad.tao.at
Running : /usr/bin/samba-tool ldapcmp --filter='whenChanged'
ldap://graz-dc-sem.ad.tao.at ldap://villach-dc-sem.ad.tao.at
Please wait.. this can take a while..
ERROR: Compare failed: -1

* Comparing [DOMAIN] context...

* Objects to be compared: 392

Comparing:
'CN=REDACTED1,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED1,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://villach-dc-sem.ad.tao.at]
    Difference in attribute values:
        mailAlt =>
[…REDACTED, GOOD…]
[…REDACTED, OUTDATED…]
    FAILED

Comparing:
'CN=REDACTED2,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED2,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://villach-dc-sem.ad.tao.at]
    Difference in attribute values:
        mailAlt =>
[…REDACTED, GOOD…]
[…REDACTED, OUTDATED…]
    FAILED

* Result for [DOMAIN]: FAILURE

SUMMARY
---------

Attributes with different values:

    mailAlt

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1630

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1566

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 176

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 21

* Result for [DNSFOREST]: SUCCESS
Running : /usr/bin/samba-tool ldapcmp --filter='whenChanged'
ldap://graz-dc-sem.ad.tao.at ldap://villach-dc-bis.ad.tao.at
Please wait.. this can take a while..
ERROR: Compare failed: -1

* Comparing [DOMAIN] context...

* Objects to be compared: 392

Comparing:
'CN=REDACTED1,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED1,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://villach-dc-sem.ad.tao.at]
    Difference in attribute values:
        mailAlt =>
[…REDACTED, GOOD…]
[…REDACTED, OUTDATED…]
    FAILED

Comparing:
'CN=REDACTED2,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED2,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://villach-dc-sem.ad.tao.at]
    Difference in attribute values:
        mailAlt =>
[…REDACTED, GOOD…]
[…REDACTED, OUTDATED…]
    FAILED

* Result for [DOMAIN]: FAILURE

SUMMARY
---------

Attributes with different values:

    mailAlt

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1630

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1566

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 176

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 21

* Result for [DNSFOREST]: SUCCESS

* Comparing [DOMAIN] context...

* Objects to be compared: 392

Comparing:
'CN=REDACTED1,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED1,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://villach-dc-bis.ad.tao.at]
    Difference in attribute values:
        mailAlt =>
[…REDACTED, GOOD…]
[…REDACTED, OUTDATED…]
    FAILED

Comparing:
'CN=REDACTED2,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED2,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://villach-dc-bis.ad.tao.at]
    Difference in attribute values:
        mailAlt =>
[…REDACTED, GOOD…]
[…REDACTED, OUTDATED…]
    FAILED

* Result for [DOMAIN]: FAILURE

SUMMARY
---------

Attributes with different values:

    mailAlt

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1630

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1566

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 176

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 21

* Result for [DNSFOREST]: SUCCESS
Running : /usr/bin/samba-tool ldapcmp --filter='whenChanged'
ldap://graz-dc-sem.ad.tao.at ldap://graz-dc-1b.ad.tao.at
Please wait.. this can take a while..
ERROR: Compare failed: -1

* Comparing [DOMAIN] context...

* Objects to be compared: 392

Comparing:
'CN=REDACTED1,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED1,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://villach-dc-sem.ad.tao.at]
    Difference in attribute values:
        mailAlt =>
[…REDACTED, GOOD…]
[…REDACTED, OUTDATED…]
    FAILED

Comparing:
'CN=REDACTED2,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED2,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://villach-dc-sem.ad.tao.at]
    Difference in attribute values:
        mailAlt =>
[…REDACTED, GOOD…]
[…REDACTED, OUTDATED…]
    FAILED

* Result for [DOMAIN]: FAILURE

SUMMARY
---------

Attributes with different values:

    mailAlt

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1630

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1566

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 176

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 21

* Result for [DNSFOREST]: SUCCESS

* Comparing [DOMAIN] context...

* Objects to be compared: 392

Comparing:
'CN=REDACTED1,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED1,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://villach-dc-bis.ad.tao.at]
    Difference in attribute values:
        mailAlt =>
[…REDACTED, GOOD…]
[…REDACTED, OUTDATED…]
    FAILED

Comparing:
'CN=REDACTED2,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED2,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://villach-dc-bis.ad.tao.at]
    Difference in attribute values:
        mailAlt =>
[…REDACTED, GOOD…]
[…REDACTED, OUTDATED…]
    FAILED

* Result for [DOMAIN]: FAILURE

SUMMARY
---------

Attributes with different values:

    mailAlt

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1630

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1566

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 176

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 21

* Result for [DNSFOREST]: SUCCESS

* Comparing [DOMAIN] context...

* Objects to be compared: 392

Comparing:
'CN=REDACTED3,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED3,CN=Users,DC=ad,DC=tao,DC=at' [ldap://graz-dc-1b.ad.tao.at]
    Difference in attribute values:
        objectClass =>
['organizationalPerson', 'person', 'taoUser',
'top', 'user']
['organizationalPerson', 'person', 'top',
'ucsUser', 'user']
    FAILED

Comparing:
'CN=REDACTED4,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED4,CN=Users,DC=ad,DC=tao,DC=at' [ldap://graz-dc-1b.ad.tao.at]
    Difference in attribute values:
        lastLogonTimestamp =>
['SMALLVALUE']
['BIGVALUE']
    FAILED

Comparing:
'CN=Enterprise Admins,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=Enterprise Admins,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-1b.ad.tao.at]
    Attributes found only in ldap://graz-dc-sem.ad.tao.at:
        msSFU30NisDomain
        gidNumber
    FAILED

Comparing:
'CN=REDACTED5,CN=Computers,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED5,CN=Computers,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-1b.ad.tao.at]
    Difference in attribute values:
        lastLogonTimestamp =>
['SMALLVALUE']
['BIGVALUE']
    FAILED

Comparing:
'CN=REDACTED6,CN=Computers,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=REDACTED6,CN=Computers,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-1b.ad.tao.at]
    Difference in attribute values:
        lastLogonTimestamp =>
['SMALLVALUE']
['BIGVALUE']
    FAILED

Comparing:
'CN=sven.schwedas,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-sem.ad.tao.at]
'CN=sven.schwedas,CN=Users,DC=ad,DC=tao,DC=at'
[ldap://graz-dc-1b.ad.tao.at]
    Difference in attribute values:
        objectClass =>
['organizationalPerson', 'person', 'posixAccount',
'taoUser', 'top', 'user']
['organizationalPerson', 'person', 'posixAccount',
'top', 'ucsUser', 'user']
    FAILED

* Result for [DOMAIN]: FAILURE

SUMMARY
---------

Attributes found only in ldap://graz-dc-sem.ad.tao.at:

    msSFU30NisDomain
    gidNumber

Attributes with different values:

    objectClass
    lastLogonTimestamp

* Comparing [CONFIGURATION] context...

* Objects to be compared: 1630

* Result for [CONFIGURATION]: SUCCESS

* Comparing [SCHEMA] context...

* Objects to be compared: 1566

* Result for [SCHEMA]: SUCCESS

* Comparing [DNSDOMAIN] context...

* Objects to be compared: 176

* Result for [DNSDOMAIN]: SUCCESS

* Comparing [DNSFOREST] context...

* Objects to be compared: 21

* Result for [DNSFOREST]: SUCCESS
.. Next check..
Running : samba-tool drs showrepl
     failures don't match
    successes don't match
     failures don't match
    successes don't match
     failures don't match
    successes don't match

Hai Sven, 

Sorry for the long wait, i had some things todo first here.

> -----Oorspronkelijk bericht-----
> Van: Sven Schwedas [mailto:sven.schwedas at tao.at] 
> Verzonden: woensdag 6 september 2017 16:06
> Aan: L.P.H. van Belle; samba at lists.samba.org
> Onderwerp: Re: [Samba] Server GC/name.dom/dom is not 
> registered with our KDC: Miscellaneous failure (see text): 
> Server (GC/name/dom at DOM) unknown
> 
> On 2017-09-06 09:28, L.P.H. van Belle wrote:
> >>> I suggest the following, move fsmo roles to villach-dc and
> >> check database replications.
> >>
> >> DB replication is already spewing errors, what am I to 
> look out for?
> > Ok, get my check db script, run it from any dc. And post me 
> the output. 
> 
> Output attached. Seems like all servers but graz-dc-sem have 
> some stuff wrong, just wonderful.
Ok base on the log, 
graz-dc-sem.ad.tao.at is the most complete/correct server. 
> 
> > https://github.com/thctlo/samba4/blob/master/samba-check-db-repl.sh
> 
> Just FYI, something's wonky with the script. Bash trips up on 
> CRLF line endings (everywhere) and 0xC2 control characters 
> (ll. 260-261) sprinkled through the file. Had to clean that 
> up before it actually ran without syntax errors.
Yes, sorry, i should have send this link. 
https://raw.githubusercontent.com/thctlo/samba4/master/samba-check-db-repl.sh 

> 
> >>> Remove the most faulty one first, graz-dc-1b, from the
> >> domain. ( check
> >>> and cleanup DNS and AD! Very important )
> >>
> >> What to check for? What to clean up?
> > Ah, thats hard to tell, this depends a bit on the errors.
> > I search/look for the left overs, first with RSAT tools and 
> samba-tool, then with ApacheStudio. 
> > I look for hostnames/UUID/ things like that, but this is 
> only done if all other options did not work. 
> > But it depends on the errors/warnings i see/get. 
> 
> None of the differences look like they could cause the 
> replication errors, looks more like they're just the results.
Faulty GUID/UUID's can and wil result in replication errors yes.. 
> 
> >>>>> Then remove a failty server and re-add it as a new
installed DC.
> >>>>> ( the good DS with FSMO)
> >>>>> First backup: /var/lib/samba/private/secrets.keytab
> >>>>> Remove the incorrect entries from keytab file with
ktutil rkt
> >>>>> /var/lib/samba/private/secrets.keytab
> >>>>> list -e -t
> >>>>
> >>>> Might as well just nuke graz-dc-sem and add a complete 
> new DC from 
> >>>> scratch, no?
> >>> No, and yes, but i preffer no, not needed (yet). 
> >>> Start with the keytab cleanup
> >>> Check the dns record if the uuid A PTR and hostnames
> >> resolve to the correct server. 
> >>> If thats the case, then no, cleanup of keytab is, i think,
> >> sufficient. 
> >>
> >> Just to confirm the order: Clean up the keytab, if that 
> doesn't work, 
> >> start removing servers?
> > Almost. Backup then ...   Cleanup keytab of the server. 
> 
> I'll try that next, unless you find something shady in the report.
> 
> >>> Yes, if its really a mess. ;-)
> >>> Then, first a an new DC, then remove, just make sure you
> >> always have 2
> >>> dc's up and running (correctly)
> >>
> >> Servers in Villach seem to run fine, thank $DEITY, so I'll 
> leave them 
> >> alone for now.
> > Ok, thats good, run the check-db script and post the 
> complete output for me. 
> > 
> >>
> >>>>> Now re-provision and you should have correct working 
> DC's again. 
> >>>>>
> >>>>> ! Before re-provisioning, make sure all OLD records
dns and
> >>>> AD are gone. 
> >>>>
> >>>> I still have undeleteable replication records from the 
> last time I 
> >>>> had to nuke a DC, nobody replied to my emails on that
issue.
> >>>
> >>> Ok, now, im out of office in about 10 min, but mail that
> >> subject for me again> I'll have a look.
> >>
> >> First message on that topic:
> >> https://lists.samba.org/archive/samba/2017-March/207429.html
> > Ok, this one, track down both uuid's, checkout which which 
> hostname belongs with these.
> 
> Going by the output of `samba-tool drs showrepl` and sam.ldb, 
> the GUIDs are as follows:
> 
> graz-dc-sem	160f5a53-5c29-4a83-aeee-6cb1dbabeed7
> graz-dc-bis	7e4973ba-4093-4523-a70f-7caa4845e34d (not in sam.ldb)
> graz-dc-1b	bcffbad8-1add-46b9-bf69-90e52c0f09ea
> villach-dc-sem	eb5f9772-cd8f-4cde-9629-f1898c94aedd
> villach-dc-bis	e1569c90-50f9-4bb5-bd85-79145e3ff6fd
> 
> villach-sem/bis want to replicate to the dead and removed 
> graz-dc-bis for some reason, but don't have its GUID in their 
> sam.ldb either.
Ok, can you do a manualy check on graz-dc-sem and villach-dc-sem 
Both command on each server. 
samba-tool dbcheck 
And 
samba-tool dbcheck --cross-nc 

Error, post them. 
> 
> > Basicly https://wiki.samba.org/index.php/Demoting_a_Samba_AD_DC
> > In based on the Demoteing wiki example. 
> > I do the same steps as shown there. Now the last three 
> pictures on the site shows where to look. 
> > At the site and Service, go through very folder, and check 
> if it is as it should be. 
> 
> I already checked it back then, and double checked it now. 
> ADUC, ADSS and DNS are all three clean and contain no 
> reference to either the old server name or its GUID.
> 
> > https://lists.samba.org/archive/samba/2017-March/207460.html
> > Between 2 and 3, there the problem starts
> >>>  – I noticed a typo in the server's `netbios name` 
> setting, corrected it, and restarted the DC. 
> > 3. Yes, for the new hostname, the old hostname is as left 
> over in the ADDC DB and/or DNS. 
> > 
> > This name GRAZ-DC-BIS, and the name with the typo. The GUID 
> for these is where to look info. 
> 
> Yeah, but where?
Now this is here ApacheStudio is very handy, but again, dont rush things here. 
With ApacheStu.. Connect to the AD, and search for the faulty GUID/UUID's 
Same for the hostname. 

Ow did i tell you to make a backup first..  ;-) make it 2. ;-) 

> 
> > Step 6. ah the point of origin of you problems with of the 
> current post? 
> 
> Hell if I know, but might be.
> 
> > 7. dnsmasq? Ok, i just hope these are not running on the DC's. 
> 
> It's running on the gateways to cache the DCs' responses and 
> handle some auxiliary stuff. ad.tao.at and all subdomains are 
> transparently forwarded to the DCs, no difference in 
> responses for _msdsc etc. between them and querying the DCs directly.
Ok, thats fine then. 
> 
> >> Last message, where I mentioned the replication bug:
> >> https://lists.samba.org/archive/samba/2017-April/207918.html
> >>
> >>> Own and if you dont use it, ApacheDirectoryStudio can help
> >> a lot with cleanup of these kind of things. 
> >>
> >> Currently I'm using the ADSI MMC snap-in, any downsides 
> compared to 
> >> ADS?
> > I dont know, never used ADS :-/
> > 
> > Track done these : GUID's, the hostname's, ipnumbers A and 
> PTR records 
> > 7e4973ba-4093-4523-a70f-7caa4845e34d
> > d613fa11-064b-4bcc-ab01-20264f870f47
> > (how, see Verifying and Creating the objectGUID Record  
> > 
> https://wiki.samba.org/index.php/Verifying_and_Creating_a_DC_DNS_Recor
> > d)
> 
> Neither of the two return any results on any DC (or on the 
> dnsmasq servers), but the GUIDs for the 4 currently existing 
> DCs work fine. This matches the data from the RSAT tools.

Now, last, which is an option. 
If one server hold a good and correct database, you can sync that one to the
other servers.
But the database must be correct. ( and backuped 2 x ) 

samba-tool dbcheck --help gives also few extra options but i never used them so
i cant tell much of these are usefull.
  --reindex             force database re-index 
  --reset-well-known-acls
  --cross-ncs  

Rowland, can you give a educated answer on these also. 

Greetz, 

Louis

On Fri, 8 Sep 2017 10:37:56 +0200
"L.P.H. van Belle via samba" <samba at lists.samba.org> wrote:

> > Neither of the two return any results on any DC (or on the 
> > dnsmasq servers), but the GUIDs for the 4 currently existing 
> > DCs work fine. This matches the data from the RSAT tools.
> 
> 
> Now, last, which is an option. 
> If one server hold a good and correct database, you can sync that one
> to the other servers. But the database must be correct. ( and
> backuped 2 x ) 
> 
> samba-tool dbcheck --help gives also few extra options but i never
> used them so i cant tell much of these are usefull.
> --reindex             force database re-index --reset-well-known-acls
>   --cross-ncs  
> 
> Rowland, can you give a educated answer on these also. 
> 
You seem to have covered everything Louis, except I noticed the mention
of 'dnsmasq servers', I think the OP needs to expand on where these are
being used.

Rowland

Thanks Rowland, 

Very appriciated. 
The dnsmasq servers are explained, these are no problem in his setup sofar i
could tell/see.


Greetz, 

Louis
 
> -----Oorspronkelijk bericht-----
> Van: samba [mailto:samba-bounces at lists.samba.org] Namens 
> Rowland Penny via samba
> Verzonden: vrijdag 8 september 2017 10:50
> Aan: samba at lists.samba.org
> Onderwerp: Re: [Samba] Server GC/name.dom/dom is not 
> registered with our KDC: Miscellaneous failure (see text): 
> Server (GC/name/dom at DOM) unknown
> 
> On Fri, 8 Sep 2017 10:37:56 +0200
> "L.P.H. van Belle via samba" <samba at lists.samba.org>
wrote:
> 
> 
> > > Neither of the two return any results on any DC (or on 
> the dnsmasq 
> > > servers), but the GUIDs for the 4 currently existing DCs 
> work fine. 
> > > This matches the data from the RSAT tools.
> > 
> > 
> > Now, last, which is an option. 
> > If one server hold a good and correct database, you can 
> sync that one 
> > to the other servers. But the database must be correct. ( 
> and backuped 
> > 2 x )
> > 
> > samba-tool dbcheck --help gives also few extra options but i never 
> > used them so i cant tell much of these are usefull.
> > --reindex             force database re-index 
> --reset-well-known-acls
> >   --cross-ncs
> > 
> > Rowland, can you give a educated answer on these also. 
> > 
> 
> You seem to have covered everything Louis, except I noticed 
> the mention of 'dnsmasq servers', I think the OP needs to 
> expand on where these are being used.
> 
> Rowland
> 
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>

On 2017-09-08 10:37, L.P.H. van Belle wrote:
> Hai Sven, 
> 
> Sorry for the long wait, i had some things todo first here.
's fine. It's not like I don't have other stuff to worry about
either.
>> -----Oorspronkelijk bericht-----
>>>>> I suggest the following, move fsmo roles to villach-dc and
>>>> check database replications.
>>>>
>>>> DB replication is already spewing errors, what am I to 
>>>> look out for?
>>> Ok, get my check db script, run it from any dc. And post me 
>>> the output. 
>>
>> Output attached. Seems like all servers but graz-dc-sem have 
>> some stuff wrong, just wonderful.
> Ok base on the log, 
> graz-dc-sem.ad.tao.at is the most complete/correct server. 
And also the one with the keytab problems. :(
>>>>>>> Then remove a failty server and re-add it as a new
installed DC.
>>>>>>> ( the good DS with FSMO)
>>>>>>> First backup: /var/lib/samba/private/secrets.keytab
>>>>>>> Remove the incorrect entries from keytab file with
ktutil rkt
>>>>>>> /var/lib/samba/private/secrets.keytab
>>>>>>> list -e -t
>>>>>>
>>>>> Start with the keytab cleanup
>>>>> Check the dns record if the uuid A PTR and hostnames
>>>>> resolve to the correct server. 
>>>>> If thats the case, then no, cleanup of keytab is, i think,
>>>>> sufficient. 
>>>>
>>>> Just to confirm the order: Clean up the keytab, if that 
>>>> doesn't work, 
>>>> start removing servers?
>>> Almost. Backup then ...   Cleanup keytab of the server. 
>>
>> I'll try that next, unless you find something shady in the report.
While going through the records with Directory Studio I noticed that the
duplicate keytab entries are mirrored by servicePrincipalName entries in
CN=GRAZ-DC-SEM,OU=Domain Controllers,DC=ad,DC=tao,DC=at

Which should I clean up, and in what order? LDAP first, then keytab?
Just LDAP?
>>>> First message on that topic:
>>>> https://lists.samba.org/archive/samba/2017-March/207429.html
>>> Ok, this one, track down both uuid's, checkout which which 
>>> hostname belongs with these.
>>
>> Going by the output of `samba-tool drs showrepl` and sam.ldb, 
>> the GUIDs are as follows:
>>
>> graz-dc-sem	160f5a53-5c29-4a83-aeee-6cb1dbabeed7
>> graz-dc-bis	7e4973ba-4093-4523-a70f-7caa4845e34d (not in sam.ldb)
>> graz-dc-1b	bcffbad8-1add-46b9-bf69-90e52c0f09ea
>> villach-dc-sem	eb5f9772-cd8f-4cde-9629-f1898c94aedd
>> villach-dc-bis	e1569c90-50f9-4bb5-bd85-79145e3ff6fd
>>
>> villach-sem/bis want to replicate to the dead and removed 
>> graz-dc-bis for some reason, but don't have its GUID in their 
>> sam.ldb either.
> Ok, can you do a manualy check on graz-dc-sem and villach-dc-sem 
> Both command on each server. 
> samba-tool dbcheck 
> And 
> samba-tool dbcheck --cross-nc 
> 
> Error, post them. 
About that… villach-dc-sem is down because its VM host went poof
yesterday. :/ Gonna get it repaired at some point next week.

On graz-dc-sem, samba-tool dbcheck shows zero errors. --cross-nc gives:
> Error: governsID CN=ucsUser,CN=Schema,CN=Configuration,DC=ad,DC=tao,DC=at
on 1.3.6.1.4.1.19414.3.2.2 already exists as an attributeId or governsId
> Error: governsID
CN=taoSharedFolder,CN=Schema,CN=Configuration,DC=ad,DC=tao,DC=at on
1.3.6.1.4.1.19414.3.2.4 already exists as an attributeId or governsId
> Error: governsID
CN=taoMailingList,CN=Schema,CN=Configuration,DC=ad,DC=tao,DC=at on
1.3.6.1.4.1.19414.3.2.3 already exists as an attributeId or governsId
Those are presumably from me fucking around with custom LDAP attributes
years ago and don't seem to be related. I hope.
>>>>>  – I noticed a typo in the server's `netbios name` 
>> setting, corrected it, and restarted the DC. 
>>> 3. Yes, for the new hostname, the old hostname is as left 
>> over in the ADDC DB and/or DNS. 
>>>
>>> This name GRAZ-DC-BIS, and the name with the typo. The GUID 
>> for these is where to look info. 
>>
>> Yeah, but where?
> Now this is here ApacheStudio is very handy, but again, dont rush things
here.
> With ApacheStu.. Connect to the AD, and search for the faulty
GUID/UUID's
> Same for the hostname. 
> 
> Ow did i tell you to make a backup first..  ;-) make it 2. ;-) 
No results for either the old name, nor the new name, nor the GUID, on
any server.

How is `drs showrepl` giving me the UUID for something that's nowhere in
LDAP?
> Now, last, which is an option. 
> If one server hold a good and correct database, you can sync that one to
the other servers.
> But the database must be correct. ( and backuped 2 x ) 
So

1. Back up graz-dc-sem
2. purge the duplicate keytab/SPN magic voodoo somehow (see above)
3. Force sync?
> samba-tool dbcheck --help gives also few extra options but i never used
them so i cant tell much of these are usefull.
>   --reindex             force database re-index 
>   --reset-well-known-acls
>   --cross-ncs  
-- 
Mit freundlichen Grüßen, / Best Regards,
Sven Schwedas, Systemadministrator
Mail/XMPP sven.schwedas at tao.at | Skype sven.schwedas
TAO Digital | Lendplatz 45 | A8020 Graz
https://www.tao-digital.at | Tel +43 680 301 7167

> -----Oorspronkelijk bericht-----
> Van: Sven Schwedas [mailto:sven.schwedas at tao.at] 
> Verzonden: vrijdag 8 september 2017 14:40
> Aan: L.P.H. van Belle; samba at lists.samba.org
> Onderwerp: Re: [Samba] Server GC/name.dom/dom is not 
> registered with our KDC: Miscellaneous failure (see text): 
> Server (GC/name/dom at DOM) unknown
> 
> On 2017-09-08 10:37, L.P.H. van Belle wrote:
> > Hai Sven,
> > 
> > Sorry for the long wait, i had some things todo first here.
> 
> 's fine. It's not like I don't have other stuff to worry about
either.
Pfew, im thinging your just waiting for me ;-) 
> 
> >> -----Oorspronkelijk bericht-----
> >>>>> I suggest the following, move fsmo roles to villach-dc
and
> >>>> check database replications.
> >>>>
> >>>> DB replication is already spewing errors, what am I to
look out
> >>>> for?
> >>> Ok, get my check db script, run it from any dc. And post me
the
> >>> output.
> >>
> >> Output attached. Seems like all servers but graz-dc-sem have some 
> >> stuff wrong, just wonderful.
> > Ok base on the log,
> > graz-dc-sem.ad.tao.at is the most complete/correct server. 
> 
> And also the one with the keytab problems. :(
Ok, yes, but wait, read on... 
> 
> >>>>>>> Then remove a failty server and re-add it as a
new
> installed DC.
> >>>>>>> ( the good DS with FSMO)
> >>>>>>> First backup:
/var/lib/samba/private/secrets.keytab
> >>>>>>> Remove the incorrect entries from keytab file
with ktutil rkt
> >>>>>>> /var/lib/samba/private/secrets.keytab
> >>>>>>> list -e -t
> >>>>>>
> >>>>> Start with the keytab cleanup
> >>>>> Check the dns record if the uuid A PTR and hostnames
resolve to
> >>>>> the correct server.
> >>>>> If thats the case, then no, cleanup of keytab is, i
think,
> >>>>> sufficient.
> >>>>
> >>>> Just to confirm the order: Clean up the keytab, if that
doesn't
> >>>> work, start removing servers?
> >>> Almost. Backup then ...   Cleanup keytab of the server. 
> >>
> >> I'll try that next, unless you find something shady in the
report.
> 
> While going through the records with Directory Studio I 
> noticed that the duplicate keytab entries are mirrored by 
> servicePrincipalName entries in CN=GRAZ-DC-SEM,OU=Domain 
> Controllers,DC=ad,DC=tao,DC=at
> 
> Which should I clean up, and in what order? LDAP first, then keytab?
> Just LDAP?
Now, only remove the incorrect ones from : servicePrincipalName 
You can do this also very easy from withing RSAT "User and Domain
manager"
Enable advanced view. 
Right klik on the computer object, properties goto tab: Attribute Editor

servicePrincipalName
edit and remove the failty ones 
And ok, exit. 

! Dont add there. (not now, first fix everything. ) 
> 
> >>>> First message on that topic:
> >>>>
https://lists.samba.org/archive/samba/2017-March/207429.html
> >>> Ok, this one, track down both uuid's, checkout which 
> which hostname 
> >>> belongs with these.
> >>
> >> Going by the output of `samba-tool drs showrepl` and sam.ldb, the 
> >> GUIDs are as follows:
> >>
> >> graz-dc-sem	160f5a53-5c29-4a83-aeee-6cb1dbabeed7
> >> graz-dc-bis	7e4973ba-4093-4523-a70f-7caa4845e34d 
> (not in sam.ldb)
> >> graz-dc-1b	bcffbad8-1add-46b9-bf69-90e52c0f09ea
> >> villach-dc-sem	eb5f9772-cd8f-4cde-9629-f1898c94aedd
> >> villach-dc-bis	e1569c90-50f9-4bb5-bd85-79145e3ff6fd
> >>
> >> villach-sem/bis want to replicate to the dead and removed 
> graz-dc-bis 
> >> for some reason, but don't have its GUID in their sam.ldb
either.
> > Ok, can you do a manualy check on graz-dc-sem and 
> villach-dc-sem Both 
> > command on each server.
> > samba-tool dbcheck
> > And
> > samba-tool dbcheck --cross-nc
> > 
> > Error, post them. 
> 
> About that? villach-dc-sem is down because its VM host went 
> poof yesterday. :/ Gonna get it repaired at some point next week.
> 
Option. 
Well this is you change... Get it, setup a clean server. 
Join the domain, Seize the FSMO roles after AD syncs are done. 
Push this AD DB to the other servers. s
> On graz-dc-sem, samba-tool dbcheck shows zero errors. 
> --cross-nc gives:
> 
> > Error: governsID 
> > CN=ucsUser,CN=Schema,CN=Configuration,DC=ad,DC=tao,DC=at on 
> > 1.3.6.1.4.1.19414.3.2.2 already exists as an attributeId or 
> governsId
> > Error: governsID 
> > CN=taoSharedFolder,CN=Schema,CN=Configuration,DC=ad,DC=tao,DC=at on 
> > 1.3.6.1.4.1.19414.3.2.4 already exists as an attributeId or 
> governsId
> > Error: governsID 
> > CN=taoMailingList,CN=Schema,CN=Configuration,DC=ad,DC=tao,DC=at on 
> > 1.3.6.1.4.1.19414.3.2.3 already exists as an attributeId or 
> governsId
> 
> Those are presumably from me fucking around with custom LDAP 
> attributes years ago and don't seem to be related. I hope.
I hope so to, i dont know that yet, but are these sill used somewere? 
If not, run safely : samba-tool dbcheck --cross-nc --fix 
If they are used, well, run it also, but prepair for extra (possible) problems.
> 
> >>>>>  ? I noticed a typo in the server's `netbios name`
> >> setting, corrected it, and restarted the DC. 
> >>> 3. Yes, for the new hostname, the old hostname is as left
> >> over in the ADDC DB and/or DNS. 
> >>>
> >>> This name GRAZ-DC-BIS, and the name with the typo. The GUID
> >> for these is where to look info. 
> >>
> >> Yeah, but where?
> > Now this is here ApacheStudio is very handy, but again, 
> dont rush things here. 
> > With ApacheStu.. Connect to the AD, and search for the faulty 
> > GUID/UUID's Same for the hostname.
> > 
> > Ow did i tell you to make a backup first..  ;-) make it 2. ;-)
> 
> No results for either the old name, nor the new name, nor the 
> GUID, on any server.
> 
> How is `drs showrepl` giving me the UUID for something that's 
> nowhere in LDAP?
Huh, yeah thats very strange, then, its still there, 
You should find it somewhere, i think your search is off, you did set the BASE
DN also?

And i forgot this one : samba-tool domain tombstones expunge
That cleanup the AD also from deleted items. 

> 
> > Now, last, which is an option. 
> > If one server hold a good and correct database, you can 
> sync that one to the other servers. 
> > But the database must be correct. ( and backuped 2 x )
> 
> So
> 
> 1. Back up graz-dc-sem
> 2. purge the duplicate keytab/SPN magic voodoo somehow (see 
> above) 
> 3. Force sync?
Now since im having a hardtime myself here with a keytab problem.  
I think a bug due GnPG2 still investigating it.  ( kerberos nfs and AES key
errors )
I suggest, we ask a dev the best way to cleanup the keytab file. 
I have several ways, but i want to know what the devs say also about this. 
And fideling around with keytabs on a member server is imo more easy then the AD
DC.


The "force sync" 
samba-tool drs replicate DEST SOURCE DN --full-sync is what i have in mind. 
So you can since the best database to the other servers. 

> 
> > samba-tool dbcheck --help gives also few extra options but 
> i never used them so i cant tell much of these are usefull. 
> >   --reindex             force database re-index 
> >   --reset-well-known-acls
> >   --cross-ncs
> 
> --
> Mit freundlichen Grüßen, / Best Regards, Sven Schwedas, 
> Systemadministrator Mail/XMPP sven.schwedas at tao.at | Skype 
> sven.schwedas TAO Digital | Lendplatz 45 | A8020 Graz 
> https://www.tao-digital.at | Tel +43 680 301 7167

Greetz, 

Louis