samba - Aug 2016 - Horrible BIND9_DLZ DNS breakage after DC replaced and samba-tool domain demote --remove-other-dead-server

On Sun, 14 Aug 2016 19:18:41 +0100
Alex Crow via samba <samba at lists.samba.org> wrote:
> 
> >
> > Ok, lets just run through this:
> > You have an NT4-style PDC
> Correct.
> > You classicupgrade this to a DC
> Yes, with BIND9_DLZ DNS backend.
> 
> > You join another computer as a DC
> >
> > At this point, have you checked that all DNS records etc are
> > correct ?
> 
> Yes, I followed the procedure on the Wiki at:
> 
> https://wiki.samba.org/index.php/Check_and_fix_DNS_entries_on_DC_joins
> 
> I setup bind as documented and start it as soon as the domain is
> joined. It works fine at this point.
> 
> In addition even after this I find essential DNS records missing, eg
> the A record for the domain only exists for the initial server, not
> the newly joined one. The same with all the SRV records.
I am going to fix this in the wiki, after you join a new DC, you need
to start and then restart Samba, this will then run 'samba_dnsupdate'
&
'samba_spnupdate'
> 
> So I issue this command to add them:
> 
> samba_dnsupdate --verbose
> 
> 
> > Is Bind9 running on both DCs at this point.
> > Is everything working as expected ?
> 
> Yes.
> 
> > You now turn off the first DC
> > You now seize all FSMO roles to the remaining DC
> 
> I've tried this in two different ways:
> 
> 1. Turn off the first DC, fsmo seize then 
> --remove-other-dead-server=<original DC name>
> 
> 2. Try to demote the first DC, fails to complete. then carry on as
> above
You can only demote a DC by running the demote command on the DC you
want to demote, that's why '--remove-other-dead-server' was written.
This is run on any DC to remove another DC, hence the 'other' part in
the argument name ;-)
  
> > Are you turning Bind9 off on the remaining DC at this point ?
> 
> After this point I've shut down the original DC.
No, are you stopping Bind that is running on the remaining DC, not the
one you have turned off.
> >
> > You run the demote command and then Bind9 will not start ?
> 
> In either of these scenarios bind9 will not start as it claims there
> are no records for my realm's domains.
Have you checked that the DNS records exist after the first DC is
removed from AD, but before you turn bind off on the remaining DC.

Rowland

On 14/08/16 19:37, Rowland Penny via samba wrote:
> On Sun, 14 Aug 2016 19:18:41 +0100
> Alex Crow via samba <samba at lists.samba.org> wrote:
>
>>> Ok, lets just run through this:
>>> You have an NT4-style PDC
>> Correct.
>>> You classicupgrade this to a DC
>> Yes, with BIND9_DLZ DNS backend.
>>
>>> You join another computer as a DC
>>>
>>> At this point, have you checked that all DNS records etc are
>>> correct ?
>> Yes, I followed the procedure on the Wiki at:
>>
>> https://wiki.samba.org/index.php/Check_and_fix_DNS_entries_on_DC_joins
>>
>> I setup bind as documented and start it as soon as the domain is
>> joined. It works fine at this point.
>>
>> In addition even after this I find essential DNS records missing, eg
>> the A record for the domain only exists for the initial server, not
>> the newly joined one. The same with all the SRV records.
> I am going to fix this in the wiki, after you join a new DC, you need
> to start and then restart Samba, this will then run
'samba_dnsupdate' &
> 'samba_spnupdate'
Is samba_spnupdate the crux of this issue then?
>
>> So I issue this command to add them:
>>
>> samba_dnsupdate --verbose
>>
>>
>>> Is Bind9 running on both DCs at this point.
>>> Is everything working as expected ?
>> Yes.
>>
>>> You now turn off the first DC
>>> You now seize all FSMO roles to the remaining DC
>> I've tried this in two different ways:
>>
>> 1. Turn off the first DC, fsmo seize then 
>> --remove-other-dead-server=<original DC name>
>>
>> 2. Try to demote the first DC, fails to complete. then carry on as
>> above
> You can only demote a DC by running the demote command on the DC you
> want to demote, that's why '--remove-other-dead-server' was
written.
> This is run on any DC to remove another DC, hence the 'other' part
in
> the argument name ;-)
I know you can only demote from the DC you want to demote - however it
failed for me with this error exactly as described on this site:

https://thingsdomakesense.wordpress.com/tag/samba-ad-dc/

Quote:

Using dc1.bales.lan as partner server for the demotion
Password for [BALES\administrator]:
Deactivating inbound replication
Asking partner server dc1.bales.lan to synchronize from us
Error while demoting, re-enabling inbound replication
ERROR(<type 'exceptions.RuntimeError'>): Error while sending a
DsReplicaSync for partion CN=Schema,CN=Configuration,DC=bales,DC=lan - (8440,
'WERR_DS_DRA_BAD_NC')
 File
"/usr/local/samba/lib64/python2.7/site-packages/samba/netcmd/domain.py",
line 786, in run
 drsuapiBind.DsReplicaSync(drsuapi_handle, 1, req1


>   
>>> Are you turning Bind9 off on the remaining DC at this point ?
>> After this point I've shut down the original DC.
> No, are you stopping Bind that is running on the remaining DC, not the
> one you have turned off.
No, I assumed bind should be running otherwise there would be no DNS for
the realm, which is why I couldn't fix anything with samba_dnsupdate as
it can't find a KDC...


>
>>> You run the demote command and then Bind9 will not start ?
>> In either of these scenarios bind9 will not start as it claims there
>> are no records for my realm's domains.
> Have you checked that the DNS records exist after the first DC is
> removed from AD, but before you turn bind off on the remaining DC.
I've done the dnsupdate on both DCs before turning off the first, and it
completes fine with after a couple of restarts of samba and bind. I'm
still not sure what I should turn off bind on the newer DC as it's
surely a requirement for the domain to function?

Many thanks

Alex



--
This message is intended only for the addressee and may contain
confidential information. Unless you are that person, you may not
disclose its contents or use it in any way and are requested to delete
the message along with any attachments and notify us immediately.
This email is not intended to, nor should it be taken to, constitute advice.
The information provided is correct to our knowledge & belief and must not
be used as a substitute for obtaining tax, regulatory, investment, legal or
any other appropriate advice.

"Transact" is operated by Integrated Financial Arrangements Ltd.
29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608
5300.
(Registered office: as above; Registered in England and Wales under
number: 3727592). Authorised and regulated by the Financial Conduct
Authority (entered on the Financial Services Register; no. 190856).

On Sun, 14 Aug 2016 20:48:04 +0100
Alex Crow via samba <samba at lists.samba.org> wrote:
> 
> 
> On 14/08/16 19:37, Rowland Penny via samba wrote:
> > On Sun, 14 Aug 2016 19:18:41 +0100
> > Alex Crow via samba <samba at lists.samba.org> wrote:
> >
> >>> Ok, lets just run through this:
> >>> You have an NT4-style PDC
> >> Correct.
> >>> You classicupgrade this to a DC
> >> Yes, with BIND9_DLZ DNS backend.
> >>
> >>> You join another computer as a DC
> >>>
> >>> At this point, have you checked that all DNS records etc are
> >>> correct ?
> >> Yes, I followed the procedure on the Wiki at:
> >>
> >>
https://wiki.samba.org/index.php/Check_and_fix_DNS_entries_on_DC_joins
> >>
> >> I setup bind as documented and start it as soon as the domain is
> >> joined. It works fine at this point.
> >>
> >> In addition even after this I find essential DNS records missing,
> >> eg the A record for the domain only exists for the initial server,
> >> not the newly joined one. The same with all the SRV records.
> > I am going to fix this in the wiki, after you join a new DC, you
> > need to start and then restart Samba, this will then run
> > 'samba_dnsupdate' & 'samba_spnupdate'
> 
> Is samba_spnupdate the crux of this issue then?
>
Probably not, what I was trying to get across was that when you first
join a machine, quite a lot of the DNS objects are not created in AD
for the second DC. When the samba binary is started it runs
'samba_dnsupdate' this uses a file to add the missing DNS objects.

So you don't need to issue the command, you just need to restart
Samba.
  
> >
> >> So I issue this command to add them:
> >>
> >> samba_dnsupdate --verbose
> >>
> >>
> >>> Is Bind9 running on both DCs at this point.
> >>> Is everything working as expected ?
> >> Yes.
> >>
> >>> You now turn off the first DC
> >>> You now seize all FSMO roles to the remaining DC
> >> I've tried this in two different ways:
> >>
> >> 1. Turn off the first DC, fsmo seize then 
> >> --remove-other-dead-server=<original DC name>
> >>
> >> 2. Try to demote the first DC, fails to complete. then carry on as
> >> above
> > You can only demote a DC by running the demote command on the DC you
> > want to demote, that's why '--remove-other-dead-server'
was written.
> > This is run on any DC to remove another DC, hence the 'other'
part
> > in the argument name ;-)
> 
> I know you can only demote from the DC you want to demote - however it
> failed for me with this error exactly as described on this site:
> 
> https://thingsdomakesense.wordpress.com/tag/samba-ad-dc/
> 
> Quote:
> 
> Using dc1.bales.lan as partner server for the demotion
> Password for [BALES\administrator]:
> Deactivating inbound replication
> Asking partner server dc1.bales.lan to synchronize from us
> Error while demoting, re-enabling inbound replication
> ERROR(<type 'exceptions.RuntimeError'>): Error while sending
a
> DsReplicaSync for partion CN=Schema,CN=Configuration,DC=bales,DC=lan
> - (8440, 'WERR_DS_DRA_BAD_NC') File
>
"/usr/local/samba/lib64/python2.7/site-packages/samba/netcmd/domain.py",
> line 786, in run drsuapiBind.DsReplicaSync(drsuapi_handle, 1, req1
> 
> 
> 
> >   
> >>> Are you turning Bind9 off on the remaining DC at this point ?
> >> After this point I've shut down the original DC.
> > No, are you stopping Bind that is running on the remaining DC, not
> > the one you have turned off.
> 
> No, I assumed bind should be running otherwise there would be no DNS
> for the realm, which is why I couldn't fix anything with
> samba_dnsupdate as it can't find a KDC...
I am fairly sure this is your problem, it should be able to find the
KDC on its own DC. Have you checked /etc/krb5.conf, /etc/hosts
and /etc/resolv.conf ?
> 
> 
> 
> >
> >>> You run the demote command and then Bind9 will not start ?
> >> In either of these scenarios bind9 will not start as it claims
> >> there are no records for my realm's domains.
> > Have you checked that the DNS records exist after the first DC is
> > removed from AD, but before you turn bind off on the remaining DC.
> 
> I've done the dnsupdate on both DCs before turning off the first, and
> it completes fine with after a couple of restarts of samba and bind.
> I'm still not sure what I should turn off bind on the newer DC as
it's
> surely a requirement for the domain to function?
> 
Yes it is, I was just making sure.

Rowland