samba - Aug 2016 - Horrible BIND9_DLZ DNS breakage after DC replaced and samba-tool domain demote --remove-other-dead-server

On Sun, 14 Aug 2016 20:48:04 +0100
Alex Crow via samba <samba at lists.samba.org> wrote:
> 
> 
> On 14/08/16 19:37, Rowland Penny via samba wrote:
> > On Sun, 14 Aug 2016 19:18:41 +0100
> > Alex Crow via samba <samba at lists.samba.org> wrote:
> >
> >>> Ok, lets just run through this:
> >>> You have an NT4-style PDC
> >> Correct.
> >>> You classicupgrade this to a DC
> >> Yes, with BIND9_DLZ DNS backend.
> >>
> >>> You join another computer as a DC
> >>>
> >>> At this point, have you checked that all DNS records etc are
> >>> correct ?
> >> Yes, I followed the procedure on the Wiki at:
> >>
> >>
https://wiki.samba.org/index.php/Check_and_fix_DNS_entries_on_DC_joins
> >>
> >> I setup bind as documented and start it as soon as the domain is
> >> joined. It works fine at this point.
> >>
> >> In addition even after this I find essential DNS records missing,
> >> eg the A record for the domain only exists for the initial server,
> >> not the newly joined one. The same with all the SRV records.
> > I am going to fix this in the wiki, after you join a new DC, you
> > need to start and then restart Samba, this will then run
> > 'samba_dnsupdate' & 'samba_spnupdate'
> 
> Is samba_spnupdate the crux of this issue then?
>
Probably not, what I was trying to get across was that when you first
join a machine, quite a lot of the DNS objects are not created in AD
for the second DC. When the samba binary is started it runs
'samba_dnsupdate' this uses a file to add the missing DNS objects.

So you don't need to issue the command, you just need to restart
Samba.
  
> >
> >> So I issue this command to add them:
> >>
> >> samba_dnsupdate --verbose
> >>
> >>
> >>> Is Bind9 running on both DCs at this point.
> >>> Is everything working as expected ?
> >> Yes.
> >>
> >>> You now turn off the first DC
> >>> You now seize all FSMO roles to the remaining DC
> >> I've tried this in two different ways:
> >>
> >> 1. Turn off the first DC, fsmo seize then 
> >> --remove-other-dead-server=<original DC name>
> >>
> >> 2. Try to demote the first DC, fails to complete. then carry on as
> >> above
> > You can only demote a DC by running the demote command on the DC you
> > want to demote, that's why '--remove-other-dead-server'
was written.
> > This is run on any DC to remove another DC, hence the 'other'
part
> > in the argument name ;-)
> 
> I know you can only demote from the DC you want to demote - however it
> failed for me with this error exactly as described on this site:
> 
> https://thingsdomakesense.wordpress.com/tag/samba-ad-dc/
> 
> Quote:
> 
> Using dc1.bales.lan as partner server for the demotion
> Password for [BALES\administrator]:
> Deactivating inbound replication
> Asking partner server dc1.bales.lan to synchronize from us
> Error while demoting, re-enabling inbound replication
> ERROR(<type 'exceptions.RuntimeError'>): Error while sending
a
> DsReplicaSync for partion CN=Schema,CN=Configuration,DC=bales,DC=lan
> - (8440, 'WERR_DS_DRA_BAD_NC') File
>
"/usr/local/samba/lib64/python2.7/site-packages/samba/netcmd/domain.py",
> line 786, in run drsuapiBind.DsReplicaSync(drsuapi_handle, 1, req1
> 
> 
> 
> >   
> >>> Are you turning Bind9 off on the remaining DC at this point ?
> >> After this point I've shut down the original DC.
> > No, are you stopping Bind that is running on the remaining DC, not
> > the one you have turned off.
> 
> No, I assumed bind should be running otherwise there would be no DNS
> for the realm, which is why I couldn't fix anything with
> samba_dnsupdate as it can't find a KDC...
I am fairly sure this is your problem, it should be able to find the
KDC on its own DC. Have you checked /etc/krb5.conf, /etc/hosts
and /etc/resolv.conf ?
> 
> 
> 
> >
> >>> You run the demote command and then Bind9 will not start ?
> >> In either of these scenarios bind9 will not start as it claims
> >> there are no records for my realm's domains.
> > Have you checked that the DNS records exist after the first DC is
> > removed from AD, but before you turn bind off on the remaining DC.
> 
> I've done the dnsupdate on both DCs before turning off the first, and
> it completes fine with after a couple of restarts of samba and bind.
> I'm still not sure what I should turn off bind on the newer DC as
it's
> surely a requirement for the domain to function?
> 
Yes it is, I was just making sure.

Rowland

> I am fairly sure this is your problem, it should be able to find the
> KDC on its own DC. Have you checked /etc/krb5.conf, /etc/hosts
> and /etc/resolv.conf ?
With the BIND server not running, and this krb5.conf:

[libdefaults]
        default_realm = SAMBA.IFA.NET
        dns_lookup_realm = false
        dns_lookup_kdc = true
~                           

samba_dnsupdate cannot find the KDC. Even if I add:

[realms]
    SAMBA4.IFA.NET {
    kdc= 172.31.0.10
}

it still complains about not finding a KDC and does not complete.

Oddly if I can use the output to figure out the DNS entries I need to
add, so I thought "ah, cool, I'll use samba-tool dns" to add them
back
in. To my great surprise, when I try to add each entry that
samba_dnsupdate says is missing, samba-tool tells me it already exists!!

/etc/hosts on the new DC:

172.31.0.10     samba4-dc-2.samba.ifa.net samba4-dc-2

also:

[root at samba4-dc-2 ~]# hostname -f
samba4-dc-2.samba.ifa.net

resolv.conf:

search samba.ifa.net. ifa.net.
nameserver 172.31.0.10


>
>> I've done the dnsupdate on both DCs before turning off the first,
and
>> it completes fine with after a couple of restarts of samba and bind.
>> I'm still not sure what I should turn off bind on the newer DC as
it's
>> surely a requirement for the domain to function?
>>
> Yes it is, I was just making sure.
>
> Rowland
Feels a bit chicken-and-egg at the moment. Is there a definitive
procedure documented for neophytes to, post-classicupgrade:

1) add an new BIND9_DLZ based DC properly
2) remove all traces of the DC used for the classicupgrade

?

Luckily I did take a VM snapshot of me 2nd DC before the DNS borked
itself this time, so I have a working domain for now.

Cheers

Alex


--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

On Sun, 14 Aug 2016 21:52:43 +0100
Alex Crow via samba <samba at lists.samba.org> wrote:
> 
> > I am fairly sure this is your problem, it should be able to find the
> > KDC on its own DC. Have you checked /etc/krb5.conf, /etc/hosts
> > and /etc/resolv.conf ?
> 
> With the BIND server not running, and this krb5.conf:
> 
> [libdefaults]
>         default_realm = SAMBA.IFA.NET
>         dns_lookup_realm = false
>         dns_lookup_kdc = true
> ~                           
> 
> samba_dnsupdate cannot find the KDC. Even if I add:
> 
> [realms]
>     SAMBA4.IFA.NET {
>     kdc= 172.31.0.10
> }
> 
Well, I don't think you can find the KDC if the DNS server isn't
running, you could try changing 'dns_lookup_kdc = true' to false 
> it still complains about not finding a KDC and does not complete.
> 
> Oddly if I can use the output to figure out the DNS entries I need to
> add, so I thought "ah, cool, I'll use samba-tool dns" to add
them back
> in. To my great surprise, when I try to add each entry that
> samba_dnsupdate says is missing, samba-tool tells me it already
> exists!!
OK, try running:

ldbedit -e nano -H /usr/local/samba/private/sam.ldb --cross-ncs
--show-binary

replace nano with your favourite editor and
'/usr/local/samba/private/sam.ldb' with the path to your sam.ldb.

You should now be able to search the entire AD and see if your entries
do exist.
> 
> /etc/hosts on the new DC:
> 
> 172.31.0.10     samba4-dc-2.samba.ifa.net samba4-dc-2
> 
> also:
> 
> [root at samba4-dc-2 ~]# hostname -f
> samba4-dc-2.samba.ifa.net
> 
> resolv.conf:
> 
> search samba.ifa.net. ifa.net.
> nameserver 172.31.0.10
> 
> 
> 
> >
> >> I've done the dnsupdate on both DCs before turning off the
first,
> >> and it completes fine with after a couple of restarts of samba and
> >> bind. I'm still not sure what I should turn off bind on the
newer
> >> DC as it's surely a requirement for the domain to function?
> >>
> > Yes it is, I was just making sure.
> >
> > Rowland
> 
> Feels a bit chicken-and-egg at the moment. Is there a definitive
> procedure documented for neophytes to, post-classicupgrade:
> 
> 1) add an new BIND9_DLZ based DC properly
> 2) remove all traces of the DC used for the classicupgrade
> 
> ?

I don't think so, most people just use the upgraded DC.

Rowland