Hi Sebastian,> On 02/06/2022 07:49, Denis CARDON wrote: >> Hi Sebastian, >> >> Le 01/06/2022 ? 23:54, Sebastian Arcus via samba a ?crit?: >>> To start with the end, until today I never realised that there are >>> specific procedures for backing up Samba AD databases - which is my >>> bad. I've always backed up /var/lib/samba and /var/cache/samba, >>> seeing as that's where Samba kept its stuff. Today I've accidentally >>> deleted /var/lib/samba, and tried to copy it back from the nightly >>> backups. Needless to say that it all went to pots, and dns is not >>> working properly any more, not matter what I try. I can provide more >>> details and logs, but first I wanted to ask if it is even worth the >>> effort? Is my backup of /var/lib/samba basically useless to restore >>> things to where they were before? >> >> like Andrew said, as for every database you should use a proper >> coherent backup for samba ldb db files. That said, from experience it >> seldom fails (backups happen in the night when not much happens). > > That's why I am a little bit stumped about what is going on here. I > tried backups of Samba databases from different days, even from the > weekend when I know there should be no activity on the server. Somehow > it doesn't make sense that they are all corrupted in the same way. > >> >> Could you try first to switch back to internal dns (if you where using >> bind-dlz), it should remove much issue with hardlinks and all. And >> turn off you bind9 on that machine. Then do a dbcheck --cross-ncs. > > That sounded promising. I followed the instructions from Samba wiki, > shutdown Bind and Samba, and ran the command - for some reason it > results in the same error: > > # samba_upgradedns --dns-backend=SAMBA_INTERNAL > Reading domain information > DNS accounts already exist > No zone file /var/lib/samba/bind-dns/dns/redacted.LAN.zone > DNS records will be automatically created > Traceback (most recent call last): > ? File "/usr/sbin/samba_upgradedns", line 348, in <module> > ??? ncname = msg[0]['nCName'][0] > KeyError: 'No such element'You don't really need this command to switch back to internal. Just remove the -dns in server services and add a dns forwarder in smb.cnof, then kill your bind9 and restart samba-ad. Denis> > I'm not sure why it is looking in /var/lib/samba/bind-dns/dns/ for a > zone file - I checked another healthy server and there is no zone file > there. > >> >> If you do a ldbsearch on the ldb files, does it crash? > > That all seems fine. I did ldbsearch on all ldb files in > /var/lib/samba/bind-dns/dns/sam.ldb.d/ - and they all displayed the > content without issues > > I'm really starting to wonder if I'm not barking up the wrong tree with > all this, and maybe it has nothing to do with corrupted backups somehow? > >> >> And if it goes back alive, please upgrade, there has been tons of >> bugfixes since samba 4.9. > > That is good advice. This server has lots of other software running > (Asterisk/exim/dovecot/spamassassin etc) - so the upgrade is a > significant effort - but I will have to get it done sooner rather than > later. I tried in the past to just upgrade Samba on its own - but ran > into a forest of issues with library version issues and compatibilities > - which broke other things on the server - so it doesn't seem like a > good idea. > > Many thanks for all the hints > >> >> >>> >>> Some basic info: >>> OS: Slackware 14.1 >>> Samba: 4.9.4 >>> Mode: Active Directory DC with file server on the same machine - only >>> one DC on domain >>> >>> Briefly, the samba_dlz plugin seems to be loading, but the logs have >>> various errors which so far I can't make sense of: >>> >>> Jun? 1 22:36:05 srv-01-op samba[11769]: >>> ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc - >>> NT_STATUS_ACCESS_DENIED >>> >>> and: >>> >>> # samba-tool dns zonelist localhost -U Administrator >>> Password for [redacted\Administrator]: >>> ERROR(runtime): uncaught exception - (9717, >>> 'WERR_DNS_ERROR_DS_UNAVAILABLE') >>> ?? File >>> "/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py", line >>> 177, in _run >>> ???? return self.run(*args, **kwargs) >>> ?? File "/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py", >>> line 670, in run >>> ???? request_filter) >>> >>> Should I just cut my loses and rebuild everything from scratch? It >>> will involve work from my part and downtime for the users, but I >>> should have really known about proper Samba AD db backups, so it is >>> what it is. >>> >>> Any pointers much appreciated. >>> >
Sebastian Arcus
2022-Jun-02 16:47 UTC
[Samba] Restoring Samba databases from /var/lib/samba
On 02/06/2022 14:53, Denis CARDON wrote:> Hi Sebastian, > >> On 02/06/2022 07:49, Denis CARDON wrote: >>> Hi Sebastian, >>> >>> Le 01/06/2022 ? 23:54, Sebastian Arcus via samba a ?crit?: >>>> To start with the end, until today I never realised that there are >>>> specific procedures for backing up Samba AD databases - which is my >>>> bad. I've always backed up /var/lib/samba and /var/cache/samba, >>>> seeing as that's where Samba kept its stuff. Today I've accidentally >>>> deleted /var/lib/samba, and tried to copy it back from the nightly >>>> backups. Needless to say that it all went to pots, and dns is not >>>> working properly any more, not matter what I try. I can provide more >>>> details and logs, but first I wanted to ask if it is even worth the >>>> effort? Is my backup of /var/lib/samba basically useless to restore >>>> things to where they were before? >>> >>> like Andrew said, as for every database you should use a proper >>> coherent backup for samba ldb db files. That said, from experience it >>> seldom fails (backups happen in the night when not much happens). >> >> That's why I am a little bit stumped about what is going on here. I >> tried backups of Samba databases from different days, even from the >> weekend when I know there should be no activity on the server. Somehow >> it doesn't make sense that they are all corrupted in the same way. >> >>> >>> Could you try first to switch back to internal dns (if you where >>> using bind-dlz), it should remove much issue with hardlinks and all. >>> And turn off you bind9 on that machine. Then do a dbcheck --cross-ncs. >> >> That sounded promising. I followed the instructions from Samba wiki, >> shutdown Bind and Samba, and ran the command - for some reason it >> results in the same error: >> >> # samba_upgradedns --dns-backend=SAMBA_INTERNAL >> Reading domain information >> DNS accounts already exist >> No zone file /var/lib/samba/bind-dns/dns/redacted.LAN.zone >> DNS records will be automatically created >> Traceback (most recent call last): >> ?? File "/usr/sbin/samba_upgradedns", line 348, in <module> >> ???? ncname = msg[0]['nCName'][0] >> KeyError: 'No such element' > > You don't really need this command to switch back to internal. Just > remove the -dns in server services and add a dns forwarder in smb.cnof, > then kill your bind9 and restart samba-ad.I just tried that. This time (with the dns records fixed, as per my previous email), switching to internal dns goes through fine. But I still get the error when trying to join the domain: "The specified server cannot perform the requested operation"> > Denis > > >> >> I'm not sure why it is looking in /var/lib/samba/bind-dns/dns/ for a >> zone file - I checked another healthy server and there is no zone file >> there. >> >>> >>> If you do a ldbsearch on the ldb files, does it crash? >> >> That all seems fine. I did ldbsearch on all ldb files in >> /var/lib/samba/bind-dns/dns/sam.ldb.d/ - and they all displayed the >> content without issues >> >> I'm really starting to wonder if I'm not barking up the wrong tree >> with all this, and maybe it has nothing to do with corrupted backups >> somehow? >> >>> >>> And if it goes back alive, please upgrade, there has been tons of >>> bugfixes since samba 4.9. >> >> That is good advice. This server has lots of other software running >> (Asterisk/exim/dovecot/spamassassin etc) - so the upgrade is a >> significant effort - but I will have to get it done sooner rather than >> later. I tried in the past to just upgrade Samba on its own - but ran >> into a forest of issues with library version issues and >> compatibilities - which broke other things on the server - so it >> doesn't seem like a good idea. >> >> Many thanks for all the hints >> >>> >>> >>>> >>>> Some basic info: >>>> OS: Slackware 14.1 >>>> Samba: 4.9.4 >>>> Mode: Active Directory DC with file server on the same machine - >>>> only one DC on domain >>>> >>>> Briefly, the samba_dlz plugin seems to be loading, but the logs have >>>> various errors which so far I can't make sense of: >>>> >>>> Jun? 1 22:36:05 srv-01-op samba[11769]: >>>> ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc - >>>> NT_STATUS_ACCESS_DENIED >>>> >>>> and: >>>> >>>> # samba-tool dns zonelist localhost -U Administrator >>>> Password for [redacted\Administrator]: >>>> ERROR(runtime): uncaught exception - (9717, >>>> 'WERR_DNS_ERROR_DS_UNAVAILABLE') >>>> ?? File >>>> "/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py", line >>>> 177, in _run >>>> ???? return self.run(*args, **kwargs) >>>> ?? File "/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py", >>>> line 670, in run >>>> ???? request_filter) >>>> >>>> Should I just cut my loses and rebuild everything from scratch? It >>>> will involve work from my part and downtime for the users, but I >>>> should have really known about proper Samba AD db backups, so it is >>>> what it is. >>>> >>>> Any pointers much appreciated. >>>> >>