samba - Jun 2022 - Restoring Samba databases from /var/lib/samba

Hi Sebastian,

Le 01/06/2022 ? 23:54, Sebastian Arcus via samba a
?crit?:
> To start with the end, until today I never realised that there are 
> specific procedures for backing up Samba AD databases - which is my bad. 
> I've always backed up /var/lib/samba and /var/cache/samba, seeing as 
> that's where Samba kept its stuff. Today I've accidentally deleted 
> /var/lib/samba, and tried to copy it back from the nightly backups. 
> Needless to say that it all went to pots, and dns is not working 
> properly any more, not matter what I try. I can provide more details and 
> logs, but first I wanted to ask if it is even worth the effort? Is my 
> backup of /var/lib/samba basically useless to restore things to where 
> they were before?
like Andrew said, as for every database you should use a proper coherent 
backup for samba ldb db files. That said, from experience it seldom 
fails (backups happen in the night when not much happens).

Could you try first to switch back to internal dns (if you where using 
bind-dlz), it should remove much issue with hardlinks and all. And turn 
off you bind9 on that machine. Then do a dbcheck --cross-ncs.

If you do a ldbsearch on the ldb files, does it crash?

And if it goes back alive, please upgrade, there has been tons of 
bugfixes since samba 4.9.

Cheers,

Denis

> 
> Some basic info:
> OS: Slackware 14.1
> Samba: 4.9.4
> Mode: Active Directory DC with file server on the same machine - only 
> one DC on domain
> 
> Briefly, the samba_dlz plugin seems to be loading, but the logs have 
> various errors which so far I can't make sense of:
> 
> Jun? 1 22:36:05 srv-01-op samba[11769]: 
> ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc - 
> NT_STATUS_ACCESS_DENIED
> 
> and:
> 
> # samba-tool dns zonelist localhost -U Administrator
> Password for [redacted\Administrator]:
> ERROR(runtime): uncaught exception - (9717, 
> 'WERR_DNS_ERROR_DS_UNAVAILABLE')
>  ? File
"/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py",
> line 177, in _run
>  ??? return self.run(*args, **kwargs)
>  ? File "/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py",
line
> 670, in run
>  ??? request_filter)
> 
> Should I just cut my loses and rebuild everything from scratch? It will 
> involve work from my part and downtime for the users, but I should have 
> really known about proper Samba AD db backups, so it is what it is.
> 
> Any pointers much appreciated.
>

Hi Denis,

On 02/06/2022 07:49, Denis CARDON wrote:
> Hi Sebastian,
> 
> Le 01/06/2022 ? 23:54, Sebastian Arcus via samba a ?crit?:
>> To start with the end, until today I never realised that there are 
>> specific procedures for backing up Samba AD databases - which is my 
>> bad. I've always backed up /var/lib/samba and /var/cache/samba,
seeing
>> as that's where Samba kept its stuff. Today I've accidentally
deleted
>> /var/lib/samba, and tried to copy it back from the nightly backups. 
>> Needless to say that it all went to pots, and dns is not working 
>> properly any more, not matter what I try. I can provide more details 
>> and logs, but first I wanted to ask if it is even worth the effort? Is 
>> my backup of /var/lib/samba basically useless to restore things to 
>> where they were before?
> 
> like Andrew said, as for every database you should use a proper coherent 
> backup for samba ldb db files. That said, from experience it seldom 
> fails (backups happen in the night when not much happens).
That's why I am a little bit stumped about what is going on here. I 
tried backups of Samba databases from different days, even from the 
weekend when I know there should be no activity on the server. Somehow 
it doesn't make sense that they are all corrupted in the same way.
> 
> Could you try first to switch back to internal dns (if you where using 
> bind-dlz), it should remove much issue with hardlinks and all. And turn 
> off you bind9 on that machine. Then do a dbcheck --cross-ncs.
That sounded promising. I followed the instructions from Samba wiki, 
shutdown Bind and Samba, and ran the command - for some reason it 
results in the same error:

# samba_upgradedns --dns-backend=SAMBA_INTERNAL
Reading domain information
DNS accounts already exist
No zone file /var/lib/samba/bind-dns/dns/redacted.LAN.zone
DNS records will be automatically created
Traceback (most recent call last):
   File "/usr/sbin/samba_upgradedns", line 348, in <module>
     ncname = msg[0]['nCName'][0]
KeyError: 'No such element'

I'm not sure why it is looking in /var/lib/samba/bind-dns/dns/ for a 
zone file - I checked another healthy server and there is no zone file 
there.
> 
> If you do a ldbsearch on the ldb files, does it crash?
That all seems fine. I did ldbsearch on all ldb files in 
/var/lib/samba/bind-dns/dns/sam.ldb.d/ - and they all displayed the 
content without issues

I'm really starting to wonder if I'm not barking up the wrong tree with 
all this, and maybe it has nothing to do with corrupted backups somehow?
> 
> And if it goes back alive, please upgrade, there has been tons of 
> bugfixes since samba 4.9.
That is good advice. This server has lots of other software running 
(Asterisk/exim/dovecot/spamassassin etc) - so the upgrade is a 
significant effort - but I will have to get it done sooner rather than 
later. I tried in the past to just upgrade Samba on its own - but ran 
into a forest of issues with library version issues and compatibilities 
- which broke other things on the server - so it doesn't seem like a 
good idea.

Many thanks for all the hints
> 
> 
>>
>> Some basic info:
>> OS: Slackware 14.1
>> Samba: 4.9.4
>> Mode: Active Directory DC with file server on the same machine - only 
>> one DC on domain
>>
>> Briefly, the samba_dlz plugin seems to be loading, but the logs have 
>> various errors which so far I can't make sense of:
>>
>> Jun? 1 22:36:05 srv-01-op samba[11769]: 
>> ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc - 
>> NT_STATUS_ACCESS_DENIED
>>
>> and:
>>
>> # samba-tool dns zonelist localhost -U Administrator
>> Password for [redacted\Administrator]:
>> ERROR(runtime): uncaught exception - (9717, 
>> 'WERR_DNS_ERROR_DS_UNAVAILABLE')
>> ?? File
"/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py",
>> line 177, in _run
>> ???? return self.run(*args, **kwargs)
>> ?? File
"/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py", line
>> 670, in run
>> ???? request_filter)
>>
>> Should I just cut my loses and rebuild everything from scratch? It 
>> will involve work from my part and downtime for the users, but I 
>> should have really known about proper Samba AD db backups, so it is 
>> what it is.
>>
>> Any pointers much appreciated.
>>