Sebastian Arcus
2022-Jun-02 16:47 UTC
[Samba] Restoring Samba databases from /var/lib/samba
On 02/06/2022 14:53, Denis CARDON wrote:> Hi Sebastian, > >> On 02/06/2022 07:49, Denis CARDON wrote: >>> Hi Sebastian, >>> >>> Le 01/06/2022 ? 23:54, Sebastian Arcus via samba a ?crit?: >>>> To start with the end, until today I never realised that there are >>>> specific procedures for backing up Samba AD databases - which is my >>>> bad. I've always backed up /var/lib/samba and /var/cache/samba, >>>> seeing as that's where Samba kept its stuff. Today I've accidentally >>>> deleted /var/lib/samba, and tried to copy it back from the nightly >>>> backups. Needless to say that it all went to pots, and dns is not >>>> working properly any more, not matter what I try. I can provide more >>>> details and logs, but first I wanted to ask if it is even worth the >>>> effort? Is my backup of /var/lib/samba basically useless to restore >>>> things to where they were before? >>> >>> like Andrew said, as for every database you should use a proper >>> coherent backup for samba ldb db files. That said, from experience it >>> seldom fails (backups happen in the night when not much happens). >> >> That's why I am a little bit stumped about what is going on here. I >> tried backups of Samba databases from different days, even from the >> weekend when I know there should be no activity on the server. Somehow >> it doesn't make sense that they are all corrupted in the same way. >> >>> >>> Could you try first to switch back to internal dns (if you where >>> using bind-dlz), it should remove much issue with hardlinks and all. >>> And turn off you bind9 on that machine. Then do a dbcheck --cross-ncs. >> >> That sounded promising. I followed the instructions from Samba wiki, >> shutdown Bind and Samba, and ran the command - for some reason it >> results in the same error: >> >> # samba_upgradedns --dns-backend=SAMBA_INTERNAL >> Reading domain information >> DNS accounts already exist >> No zone file /var/lib/samba/bind-dns/dns/redacted.LAN.zone >> DNS records will be automatically created >> Traceback (most recent call last): >> ?? File "/usr/sbin/samba_upgradedns", line 348, in <module> >> ???? ncname = msg[0]['nCName'][0] >> KeyError: 'No such element' > > You don't really need this command to switch back to internal. Just > remove the -dns in server services and add a dns forwarder in smb.cnof, > then kill your bind9 and restart samba-ad.I just tried that. This time (with the dns records fixed, as per my previous email), switching to internal dns goes through fine. But I still get the error when trying to join the domain: "The specified server cannot perform the requested operation"> > Denis > > >> >> I'm not sure why it is looking in /var/lib/samba/bind-dns/dns/ for a >> zone file - I checked another healthy server and there is no zone file >> there. >> >>> >>> If you do a ldbsearch on the ldb files, does it crash? >> >> That all seems fine. I did ldbsearch on all ldb files in >> /var/lib/samba/bind-dns/dns/sam.ldb.d/ - and they all displayed the >> content without issues >> >> I'm really starting to wonder if I'm not barking up the wrong tree >> with all this, and maybe it has nothing to do with corrupted backups >> somehow? >> >>> >>> And if it goes back alive, please upgrade, there has been tons of >>> bugfixes since samba 4.9. >> >> That is good advice. This server has lots of other software running >> (Asterisk/exim/dovecot/spamassassin etc) - so the upgrade is a >> significant effort - but I will have to get it done sooner rather than >> later. I tried in the past to just upgrade Samba on its own - but ran >> into a forest of issues with library version issues and >> compatibilities - which broke other things on the server - so it >> doesn't seem like a good idea. >> >> Many thanks for all the hints >> >>> >>> >>>> >>>> Some basic info: >>>> OS: Slackware 14.1 >>>> Samba: 4.9.4 >>>> Mode: Active Directory DC with file server on the same machine - >>>> only one DC on domain >>>> >>>> Briefly, the samba_dlz plugin seems to be loading, but the logs have >>>> various errors which so far I can't make sense of: >>>> >>>> Jun? 1 22:36:05 srv-01-op samba[11769]: >>>> ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc - >>>> NT_STATUS_ACCESS_DENIED >>>> >>>> and: >>>> >>>> # samba-tool dns zonelist localhost -U Administrator >>>> Password for [redacted\Administrator]: >>>> ERROR(runtime): uncaught exception - (9717, >>>> 'WERR_DNS_ERROR_DS_UNAVAILABLE') >>>> ?? File >>>> "/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py", line >>>> 177, in _run >>>> ???? return self.run(*args, **kwargs) >>>> ?? File "/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py", >>>> line 670, in run >>>> ???? request_filter) >>>> >>>> Should I just cut my loses and rebuild everything from scratch? It >>>> will involve work from my part and downtime for the users, but I >>>> should have really known about proper Samba AD db backups, so it is >>>> what it is. >>>> >>>> Any pointers much appreciated. >>>> >>
On Thu, 2022-06-02 at 17:47 +0100, Sebastian Arcus via samba wrote:> On 02/06/2022 14:53, Denis CARDON wrote: > > Hi Sebastian, > > > > > On 02/06/2022 07:49, Denis CARDON wrote: > > > > Hi Sebastian, > > > > > > > > Le 01/06/2022 ? 23:54, Sebastian Arcus via samba a ?crit : > > > > > To start with the end, until today I never realised that > > > > > there are > > > > > specific procedures for backing up Samba AD databases - which > > > > > is my > > > > > bad. I've always backed up /var/lib/samba and > > > > > /var/cache/samba, > > > > > seeing as that's where Samba kept its stuff. Today I've > > > > > accidentally > > > > > deleted /var/lib/samba, and tried to copy it back from the > > > > > nightly > > > > > backups. Needless to say that it all went to pots, and dns is > > > > > not > > > > > working properly any more, not matter what I try. I can > > > > > provide more > > > > > details and logs, but first I wanted to ask if it is even > > > > > worth the > > > > > effort? Is my backup of /var/lib/samba basically useless to > > > > > restore > > > > > things to where they were before? > > > > > > > > like Andrew said, as for every database you should use a > > > > proper > > > > coherent backup for samba ldb db files. That said, from > > > > experience it > > > > seldom fails (backups happen in the night when not much > > > > happens). > > > > > > That's why I am a little bit stumped about what is going on here. > > > I > > > tried backups of Samba databases from different days, even from > > > the > > > weekend when I know there should be no activity on the server. > > > Somehow > > > it doesn't make sense that they are all corrupted in the same > > > way. > > > > > > > Could you try first to switch back to internal dns (if you > > > > where > > > > using bind-dlz), it should remove much issue with hardlinks and > > > > all. > > > > And turn off you bind9 on that machine. Then do a dbcheck -- > > > > cross-ncs. > > > > > > That sounded promising. I followed the instructions from Samba > > > wiki, > > > shutdown Bind and Samba, and ran the command - for some reason > > > it > > > results in the same error: > > > > > > # samba_upgradedns --dns-backend=SAMBA_INTERNAL > > > Reading domain information > > > DNS accounts already exist > > > No zone file /var/lib/samba/bind-dns/dns/redacted.LAN.zone > > > DNS records will be automatically created > > > Traceback (most recent call last): > > > File "/usr/sbin/samba_upgradedns", line 348, in <module> > > > ncname = msg[0]['nCName'][0] > > > KeyError: 'No such element' > > > > You don't really need this command to switch back to internal. > > Just > > remove the -dns in server services and add a dns forwarder in > > smb.cnof, > > then kill your bind9 and restart samba-ad. > > I just tried that. This time (with the dns records fixed, as per my > previous email), switching to internal dns goes through fine. But I > still get the error when trying to join the domain: > > "The specified server cannot perform the requested operation"What are you trying to join ? The DC ? If so you do not join a DC from backups. I know it is a bit late now, but you should have used 'samba-tool domain backup'. This is used to backup the domain and not an individual DC, you then use the same tool to restore the domain. Copying individual files or directories is likely to cause problems, it may work, but you may get parts of the database that are older than others. If you are trying to restore in this way, then trying to join the DC again isn't likely to work. you need an existing DC to join to. Rowland