Well done! That all looks quite sensible. Perhaps write it up as a wiki page? Glad things worked out OK. The --rpc-server-ip bit will force updates over a protocol that is less reliant on DNS working, and that will help a lot (which is why it was implemented, it is also in there automatically as a fallback). Andrew Bartlett On Tue, 2021-06-22 at 09:43 +0000, Chris Puttick via samba wrote:> Ok, so we're not sure whether running the samba_upgradedns was part > of the fix, but for those who encounter similar scenarios, these > steps did seem to resolve it (over time): > > Recreate the zone with > > samba-tool dns zonecreate A-DC domain.local -U admin.username > > (where in this organisation's case A-DC was what we think of in old- > fashioned terms as the PDC e.g. has the FSMO roles - may or may not > be relevant) > > The zone starts to populate itself fairly quickly with stuff from the > LAN. > > On the DCs restart samba > > (service samba-ad-dc restart) > > That gets the zone recreated on the DCs, and a little while later the > DCs appear in as records in the zone, checking with > > samba-tool dns query this-DC domain.local @ ALL -U admin.username > > Somehow or other, probably with some variant of samba_dnsupdate we > managed to get one of the remote DCs A record to get updated more > quickly. Waiting (impatiently) for the others took an age (maybe 15 > minutes ;) ) to appear. > > The one that got forced had the command run *before* samba was > restarted, with the form that resulted in most apparent activity (but > without the domain zone being recreated on the DC) including said DC > appearing as a record. > > samba_dnsupdate --verbose --rpc-server-ip ip.of.a-.dc --all-names > > Again noting this may or may not be a factor, as other DCs which > didn't have any samba_dnsupdate commands run eventually appeared in a > seemingly random length of time (where a DC on each remote site was > shutdown for safety, and booted up about the same time, with 10+ > minutes difference in when their A records got created. > > So while the idea that the AD zone should not be deletable without > some complex, preferably CLI only, process is a great one, recovery > from the deletion seems relatively painless. > > Cheers > > Chris > > ----- Original Message ----- > From: "Andrew Bartlett" <abartlet at samba.org> > To: "Chris Puttick" <chris.puttick at cp1associates.net> > Cc: "samba" <samba at lists.samba.org> > Sent: Tuesday, 22 June, 2021 08:40:45 > Subject: Re: [Samba] Accidental zone deletion > > On Tue, 2021-06-22 at 07:16 +0000, Chris Puttick via samba wrote: > > Thanks for the response, about what we feared. For interest on one > > of > > the DCs we tried > > > > samba_upgradedns --dns-backend=SAMBA_INTERNAL --migrate=no > > > > and got the response > > > > # samba_upgradedns --dns-backend=SAMBA_INTERNAL --migrate=no > > Reading domain information > > DNS accounts already exist > > No zone file /var/lib/samba/private/dns/OXARCH.LOCAL.zone > > DNS records will be automatically created > > DNS partitions already exist > > Finished upgrading DNS > > This may have rebuilt a very minimal DNS, which may be a good thing. > > > Found the comment about "no zone file" interesting because that > > directory doesn't exist and AFAIK has never existed (Ubuntu 18.04 > > running Samba 4.7.6-Ubuntu). > > Correct. That tool does a number of different things, but one thing > it > was built for originally was upgrading from a file-based zone into > the > in-directory zone, hence that check and message. > > Andrew Bartlett > > > > ----- Original Message ----- > > From: "Andrew Bartlett" <abartlet at samba.org> > > To: "Chris Puttick" <chris.puttick at cp1associates.net>, "samba" < > > samba at lists.samba.org> > > Sent: Tuesday, 22 June, 2021 06:47:29 > > Subject: Re: [Samba] Accidental zone deletion > > > > On Tue, 2021-06-22 at 05:29 +0000, Chris Puttick via samba wrote: > > > Hi > > > > > > We have a situation where an MS admin used the AD utilities to > > > tidy > > > up an neighbouring (MS-based) domain but was attached to the > > > wrong > > > DC > > > and deleted the wrongdomain.local zone file (which is apparently > > > a > > > bit of a thing in MS circles); by the time said admin realised > > > the > > > deletion had replicated across DCs on all sites. How do we > > > recreate > > > it, in partiular the contents? Hoping the answer is "just > > > manually > > > create the zone and it'll repopulate". > > > > > > Any suggestions welcomed... > > > > I assume of course you mean the zone in a Samba AD DC, not a simple > > .zone file. > > > > This has happened, and yes, I do think we should prevent it at the > > database level, as nobody ever really means to do that. Last time > > that > > happened we helped a client jury-rig up a backup of the sam.ldb > > into > > BIND9-DLZ (so only DNS used the old data), allowing service to > > somewhat > > continue while things were fixed back up. > > > > However, I'm sorry to say it won't just be regenerated, while Samba > > will try and re-register itself every now and then, I wouldn't > > count > > on > > it getting back the way you found it fast. > > > > How are your backups? > > > > Andrew Bartlett > > > > -- > > Andrew Bartlett (he/him) https://samba.org/~abartlet/ > > Samba Team Member (since 2001) https://samba.org > > Samba Team Lead, Catalyst IT > > https://catalyst.net.nz/services/samba > > > > Samba Development and Support, Catalyst IT - Expert Open Source > > Solutions > > > -- > Andrew Bartlett (he/him) https://samba.org/~abartlet/ > Samba Team Member (since 2001) https://samba.org > Samba Team Lead, Catalyst IT https://catalyst.net.nz/services/samba > > Samba Development and Support, Catalyst IT - Expert Open Source > Solutions >-- Andrew Bartlett (he/him) https://samba.org/~abartlet/ Samba Team Member (since 2001) https://samba.org Samba Team Lead, Catalyst IT https://catalyst.net.nz/services/samba Samba Development and Support, Catalyst IT - Expert Open Source Solutions
We've got some further refinements and happy to write it up as a wiki page - just need to get the capture answer so I can sign up to edit the wiki. NB while very tempted to blow the customer's zone away and test the fix, be better if someone has a lab setup they can verify it on :) ----- Original Message ----- From: "Andrew Bartlett" <abartlet at samba.org> To: "Chris Puttick" <chris.puttick at cp1associates.net> Cc: "samba" <samba at lists.samba.org> Sent: Tuesday, 22 June, 2021 10:56:07 Subject: Re: [Samba] Accidental zone deletion Well done! That all looks quite sensible. Perhaps write it up as a wiki page? Glad things worked out OK. The --rpc-server-ip bit will force updates over a protocol that is less reliant on DNS working, and that will help a lot (which is why it was implemented, it is also in there automatically as a fallback). Andrew Bartlett On Tue, 2021-06-22 at 09:43 +0000, Chris Puttick via samba wrote:> Ok, so we're not sure whether running the samba_upgradedns was part > of the fix, but for those who encounter similar scenarios, these > steps did seem to resolve it (over time): > > Recreate the zone with > > samba-tool dns zonecreate A-DC domain.local -U admin.username > > (where in this organisation's case A-DC was what we think of in old- > fashioned terms as the PDC e.g. has the FSMO roles - may or may not > be relevant) > > The zone starts to populate itself fairly quickly with stuff from the > LAN. > > On the DCs restart samba > > (service samba-ad-dc restart) > > That gets the zone recreated on the DCs, and a little while later the > DCs appear in as records in the zone, checking with > > samba-tool dns query this-DC domain.local @ ALL -U admin.username > > Somehow or other, probably with some variant of samba_dnsupdate we > managed to get one of the remote DCs A record to get updated more > quickly. Waiting (impatiently) for the others took an age (maybe 15 > minutes ;) ) to appear. > > The one that got forced had the command run *before* samba was > restarted, with the form that resulted in most apparent activity (but > without the domain zone being recreated on the DC) including said DC > appearing as a record. > > samba_dnsupdate --verbose --rpc-server-ip ip.of.a-.dc --all-names > > Again noting this may or may not be a factor, as other DCs which > didn't have any samba_dnsupdate commands run eventually appeared in a > seemingly random length of time (where a DC on each remote site was > shutdown for safety, and booted up about the same time, with 10+ > minutes difference in when their A records got created. > > So while the idea that the AD zone should not be deletable without > some complex, preferably CLI only, process is a great one, recovery > from the deletion seems relatively painless. > > Cheers > > Chris > > ----- Original Message ----- > From: "Andrew Bartlett" <abartlet at samba.org> > To: "Chris Puttick" <chris.puttick at cp1associates.net> > Cc: "samba" <samba at lists.samba.org> > Sent: Tuesday, 22 June, 2021 08:40:45 > Subject: Re: [Samba] Accidental zone deletion > > On Tue, 2021-06-22 at 07:16 +0000, Chris Puttick via samba wrote: > > Thanks for the response, about what we feared. For interest on one > > of > > the DCs we tried > > > > samba_upgradedns --dns-backend=SAMBA_INTERNAL --migrate=no > > > > and got the response > > > > # samba_upgradedns --dns-backend=SAMBA_INTERNAL --migrate=no > > Reading domain information > > DNS accounts already exist > > No zone file /var/lib/samba/private/dns/OXARCH.LOCAL.zone > > DNS records will be automatically created > > DNS partitions already exist > > Finished upgrading DNS > > This may have rebuilt a very minimal DNS, which may be a good thing. > > > Found the comment about "no zone file" interesting because that > > directory doesn't exist and AFAIK has never existed (Ubuntu 18.04 > > running Samba 4.7.6-Ubuntu). > > Correct. That tool does a number of different things, but one thing > it > was built for originally was upgrading from a file-based zone into > the > in-directory zone, hence that check and message. > > Andrew Bartlett > > > > ----- Original Message ----- > > From: "Andrew Bartlett" <abartlet at samba.org> > > To: "Chris Puttick" <chris.puttick at cp1associates.net>, "samba" < > > samba at lists.samba.org> > > Sent: Tuesday, 22 June, 2021 06:47:29 > > Subject: Re: [Samba] Accidental zone deletion > > > > On Tue, 2021-06-22 at 05:29 +0000, Chris Puttick via samba wrote: > > > Hi > > > > > > We have a situation where an MS admin used the AD utilities to > > > tidy > > > up an neighbouring (MS-based) domain but was attached to the > > > wrong > > > DC > > > and deleted the wrongdomain.local zone file (which is apparently > > > a > > > bit of a thing in MS circles); by the time said admin realised > > > the > > > deletion had replicated across DCs on all sites. How do we > > > recreate > > > it, in partiular the contents? Hoping the answer is "just > > > manually > > > create the zone and it'll repopulate". > > > > > > Any suggestions welcomed... > > > > I assume of course you mean the zone in a Samba AD DC, not a simple > > .zone file. > > > > This has happened, and yes, I do think we should prevent it at the > > database level, as nobody ever really means to do that. Last time > > that > > happened we helped a client jury-rig up a backup of the sam.ldb > > into > > BIND9-DLZ (so only DNS used the old data), allowing service to > > somewhat > > continue while things were fixed back up. > > > > However, I'm sorry to say it won't just be regenerated, while Samba > > will try and re-register itself every now and then, I wouldn't > > count > > on > > it getting back the way you found it fast. > > > > How are your backups? > > > > Andrew Bartlett > > > > -- > > Andrew Bartlett (he/him) https://samba.org/~abartlet/ > > Samba Team Member (since 2001) https://samba.org > > Samba Team Lead, Catalyst IT > > https://catalyst.net.nz/services/samba > > > > Samba Development and Support, Catalyst IT - Expert Open Source > > Solutions > > > -- > Andrew Bartlett (he/him) https://samba.org/~abartlet/ > Samba Team Member (since 2001) https://samba.org > Samba Team Lead, Catalyst IT https://catalyst.net.nz/services/samba > > Samba Development and Support, Catalyst IT - Expert Open Source > Solutions >-- Andrew Bartlett (he/him) https://samba.org/~abartlet/ Samba Team Member (since 2001) https://samba.org Samba Team Lead, Catalyst IT https://catalyst.net.nz/services/samba Samba Development and Support, Catalyst IT - Expert Open Source Solutions
Wiki page (with refinements to the content of "as we're doing it" previous message) is done: https://wiki.samba.org/index.php/Deleted_AD_zone ----- Original Message ----- From: "Andrew Bartlett" <abartlet at samba.org> To: "Chris Puttick" <chris.puttick at cp1associates.net> Cc: "samba" <samba at lists.samba.org> Sent: Tuesday, 22 June, 2021 10:56:07 Subject: Re: [Samba] Accidental zone deletion Well done! That all looks quite sensible. Perhaps write it up as a wiki page? Glad things worked out OK. The --rpc-server-ip bit will force updates over a protocol that is less reliant on DNS working, and that will help a lot (which is why it was implemented, it is also in there automatically as a fallback). Andrew Bartlett On Tue, 2021-06-22 at 09:43 +0000, Chris Puttick via samba wrote:> Ok, so we're not sure whether running the samba_upgradedns was part > of the fix, but for those who encounter similar scenarios, these > steps did seem to resolve it (over time): > > Recreate the zone with > > samba-tool dns zonecreate A-DC domain.local -U admin.username > > (where in this organisation's case A-DC was what we think of in old- > fashioned terms as the PDC e.g. has the FSMO roles - may or may not > be relevant) > > The zone starts to populate itself fairly quickly with stuff from the > LAN. > > On the DCs restart samba > > (service samba-ad-dc restart) > > That gets the zone recreated on the DCs, and a little while later the > DCs appear in as records in the zone, checking with > > samba-tool dns query this-DC domain.local @ ALL -U admin.username > > Somehow or other, probably with some variant of samba_dnsupdate we > managed to get one of the remote DCs A record to get updated more > quickly. Waiting (impatiently) for the others took an age (maybe 15 > minutes ;) ) to appear. > > The one that got forced had the command run *before* samba was > restarted, with the form that resulted in most apparent activity (but > without the domain zone being recreated on the DC) including said DC > appearing as a record. > > samba_dnsupdate --verbose --rpc-server-ip ip.of.a-.dc --all-names > > Again noting this may or may not be a factor, as other DCs which > didn't have any samba_dnsupdate commands run eventually appeared in a > seemingly random length of time (where a DC on each remote site was > shutdown for safety, and booted up about the same time, with 10+ > minutes difference in when their A records got created. > > So while the idea that the AD zone should not be deletable without > some complex, preferably CLI only, process is a great one, recovery > from the deletion seems relatively painless. > > Cheers > > Chris > > ----- Original Message ----- > From: "Andrew Bartlett" <abartlet at samba.org> > To: "Chris Puttick" <chris.puttick at cp1associates.net> > Cc: "samba" <samba at lists.samba.org> > Sent: Tuesday, 22 June, 2021 08:40:45 > Subject: Re: [Samba] Accidental zone deletion > > On Tue, 2021-06-22 at 07:16 +0000, Chris Puttick via samba wrote: > > Thanks for the response, about what we feared. For interest on one > > of > > the DCs we tried > > > > samba_upgradedns --dns-backend=SAMBA_INTERNAL --migrate=no > > > > and got the response > > > > # samba_upgradedns --dns-backend=SAMBA_INTERNAL --migrate=no > > Reading domain information > > DNS accounts already exist > > No zone file /var/lib/samba/private/dns/OXARCH.LOCAL.zone > > DNS records will be automatically created > > DNS partitions already exist > > Finished upgrading DNS > > This may have rebuilt a very minimal DNS, which may be a good thing. > > > Found the comment about "no zone file" interesting because that > > directory doesn't exist and AFAIK has never existed (Ubuntu 18.04 > > running Samba 4.7.6-Ubuntu). > > Correct. That tool does a number of different things, but one thing > it > was built for originally was upgrading from a file-based zone into > the > in-directory zone, hence that check and message. > > Andrew Bartlett > > > > ----- Original Message ----- > > From: "Andrew Bartlett" <abartlet at samba.org> > > To: "Chris Puttick" <chris.puttick at cp1associates.net>, "samba" < > > samba at lists.samba.org> > > Sent: Tuesday, 22 June, 2021 06:47:29 > > Subject: Re: [Samba] Accidental zone deletion > > > > On Tue, 2021-06-22 at 05:29 +0000, Chris Puttick via samba wrote: > > > Hi > > > > > > We have a situation where an MS admin used the AD utilities to > > > tidy > > > up an neighbouring (MS-based) domain but was attached to the > > > wrong > > > DC > > > and deleted the wrongdomain.local zone file (which is apparently > > > a > > > bit of a thing in MS circles); by the time said admin realised > > > the > > > deletion had replicated across DCs on all sites. How do we > > > recreate > > > it, in partiular the contents? Hoping the answer is "just > > > manually > > > create the zone and it'll repopulate". > > > > > > Any suggestions welcomed... > > > > I assume of course you mean the zone in a Samba AD DC, not a simple > > .zone file. > > > > This has happened, and yes, I do think we should prevent it at the > > database level, as nobody ever really means to do that. Last time > > that > > happened we helped a client jury-rig up a backup of the sam.ldb > > into > > BIND9-DLZ (so only DNS used the old data), allowing service to > > somewhat > > continue while things were fixed back up. > > > > However, I'm sorry to say it won't just be regenerated, while Samba > > will try and re-register itself every now and then, I wouldn't > > count > > on > > it getting back the way you found it fast. > > > > How are your backups? > > > > Andrew Bartlett > > > > -- > > Andrew Bartlett (he/him) https://samba.org/~abartlet/ > > Samba Team Member (since 2001) https://samba.org > > Samba Team Lead, Catalyst IT > > https://catalyst.net.nz/services/samba > > > > Samba Development and Support, Catalyst IT - Expert Open Source > > Solutions > > > -- > Andrew Bartlett (he/him) https://samba.org/~abartlet/ > Samba Team Member (since 2001) https://samba.org > Samba Team Lead, Catalyst IT https://catalyst.net.nz/services/samba > > Samba Development and Support, Catalyst IT - Expert Open Source > Solutions >-- Andrew Bartlett (he/him) https://samba.org/~abartlet/ Samba Team Member (since 2001) https://samba.org Samba Team Lead, Catalyst IT https://catalyst.net.nz/services/samba Samba Development and Support, Catalyst IT - Expert Open Source Solutions