samba - Jun 2022 - Restoring Samba databases from /var/lib/samba

Hi Denis,

On 02/06/2022 07:49, Denis CARDON wrote:
> Hi Sebastian,
> 
> Le 01/06/2022 ? 23:54, Sebastian Arcus via samba a ?crit?:
>> To start with the end, until today I never realised that there are 
>> specific procedures for backing up Samba AD databases - which is my 
>> bad. I've always backed up /var/lib/samba and /var/cache/samba,
seeing
>> as that's where Samba kept its stuff. Today I've accidentally
deleted
>> /var/lib/samba, and tried to copy it back from the nightly backups. 
>> Needless to say that it all went to pots, and dns is not working 
>> properly any more, not matter what I try. I can provide more details 
>> and logs, but first I wanted to ask if it is even worth the effort? Is 
>> my backup of /var/lib/samba basically useless to restore things to 
>> where they were before?
> 
> like Andrew said, as for every database you should use a proper coherent 
> backup for samba ldb db files. That said, from experience it seldom 
> fails (backups happen in the night when not much happens).
That's why I am a little bit stumped about what is going on here. I 
tried backups of Samba databases from different days, even from the 
weekend when I know there should be no activity on the server. Somehow 
it doesn't make sense that they are all corrupted in the same way.
> 
> Could you try first to switch back to internal dns (if you where using 
> bind-dlz), it should remove much issue with hardlinks and all. And turn 
> off you bind9 on that machine. Then do a dbcheck --cross-ncs.
That sounded promising. I followed the instructions from Samba wiki, 
shutdown Bind and Samba, and ran the command - for some reason it 
results in the same error:

# samba_upgradedns --dns-backend=SAMBA_INTERNAL
Reading domain information
DNS accounts already exist
No zone file /var/lib/samba/bind-dns/dns/redacted.LAN.zone
DNS records will be automatically created
Traceback (most recent call last):
   File "/usr/sbin/samba_upgradedns", line 348, in <module>
     ncname = msg[0]['nCName'][0]
KeyError: 'No such element'

I'm not sure why it is looking in /var/lib/samba/bind-dns/dns/ for a 
zone file - I checked another healthy server and there is no zone file 
there.
> 
> If you do a ldbsearch on the ldb files, does it crash?
That all seems fine. I did ldbsearch on all ldb files in 
/var/lib/samba/bind-dns/dns/sam.ldb.d/ - and they all displayed the 
content without issues

I'm really starting to wonder if I'm not barking up the wrong tree with 
all this, and maybe it has nothing to do with corrupted backups somehow?
> 
> And if it goes back alive, please upgrade, there has been tons of 
> bugfixes since samba 4.9.
That is good advice. This server has lots of other software running 
(Asterisk/exim/dovecot/spamassassin etc) - so the upgrade is a 
significant effort - but I will have to get it done sooner rather than 
later. I tried in the past to just upgrade Samba on its own - but ran 
into a forest of issues with library version issues and compatibilities 
- which broke other things on the server - so it doesn't seem like a 
good idea.

Many thanks for all the hints
> 
> 
>>
>> Some basic info:
>> OS: Slackware 14.1
>> Samba: 4.9.4
>> Mode: Active Directory DC with file server on the same machine - only 
>> one DC on domain
>>
>> Briefly, the samba_dlz plugin seems to be loading, but the logs have 
>> various errors which so far I can't make sense of:
>>
>> Jun? 1 22:36:05 srv-01-op samba[11769]: 
>> ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc - 
>> NT_STATUS_ACCESS_DENIED
>>
>> and:
>>
>> # samba-tool dns zonelist localhost -U Administrator
>> Password for [redacted\Administrator]:
>> ERROR(runtime): uncaught exception - (9717, 
>> 'WERR_DNS_ERROR_DS_UNAVAILABLE')
>> ?? File
"/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py",
>> line 177, in _run
>> ???? return self.run(*args, **kwargs)
>> ?? File
"/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py", line
>> 670, in run
>> ???? request_filter)
>>
>> Should I just cut my loses and rebuild everything from scratch? It 
>> will involve work from my part and downtime for the users, but I 
>> should have really known about proper Samba AD db backups, so it is 
>> what it is.
>>
>> Any pointers much appreciated.
>>

Hi Sebastian,
> On 02/06/2022 07:49, Denis CARDON wrote:
>> Hi Sebastian,
>>
>> Le 01/06/2022 ? 23:54, Sebastian Arcus via samba a ?crit?:
>>> To start with the end, until today I never realised that there are 
>>> specific procedures for backing up Samba AD databases - which is my
>>> bad. I've always backed up /var/lib/samba and /var/cache/samba,
>>> seeing as that's where Samba kept its stuff. Today I've
accidentally
>>> deleted /var/lib/samba, and tried to copy it back from the nightly 
>>> backups. Needless to say that it all went to pots, and dns is not 
>>> working properly any more, not matter what I try. I can provide
more
>>> details and logs, but first I wanted to ask if it is even worth the
>>> effort? Is my backup of /var/lib/samba basically useless to restore
>>> things to where they were before?
>>
>> like Andrew said, as for every database you should use a proper 
>> coherent backup for samba ldb db files. That said, from experience it 
>> seldom fails (backups happen in the night when not much happens).
> 
> That's why I am a little bit stumped about what is going on here. I 
> tried backups of Samba databases from different days, even from the 
> weekend when I know there should be no activity on the server. Somehow 
> it doesn't make sense that they are all corrupted in the same way.
> 
>>
>> Could you try first to switch back to internal dns (if you where using 
>> bind-dlz), it should remove much issue with hardlinks and all. And 
>> turn off you bind9 on that machine. Then do a dbcheck --cross-ncs.
> 
> That sounded promising. I followed the instructions from Samba wiki, 
> shutdown Bind and Samba, and ran the command - for some reason it 
> results in the same error:
> 
> # samba_upgradedns --dns-backend=SAMBA_INTERNAL
> Reading domain information
> DNS accounts already exist
> No zone file /var/lib/samba/bind-dns/dns/redacted.LAN.zone
> DNS records will be automatically created
> Traceback (most recent call last):
>  ? File "/usr/sbin/samba_upgradedns", line 348, in <module>
>  ??? ncname = msg[0]['nCName'][0]
> KeyError: 'No such element'
You don't really need this command to switch back to internal. Just 
remove the -dns in server services and add a dns forwarder in smb.cnof, 
then kill your bind9 and restart samba-ad.

Denis

> 
> I'm not sure why it is looking in /var/lib/samba/bind-dns/dns/ for a 
> zone file - I checked another healthy server and there is no zone file 
> there.
> 
>>
>> If you do a ldbsearch on the ldb files, does it crash?
> 
> That all seems fine. I did ldbsearch on all ldb files in 
> /var/lib/samba/bind-dns/dns/sam.ldb.d/ - and they all displayed the 
> content without issues
> 
> I'm really starting to wonder if I'm not barking up the wrong tree
with
> all this, and maybe it has nothing to do with corrupted backups somehow?
> 
>>
>> And if it goes back alive, please upgrade, there has been tons of 
>> bugfixes since samba 4.9.
> 
> That is good advice. This server has lots of other software running 
> (Asterisk/exim/dovecot/spamassassin etc) - so the upgrade is a 
> significant effort - but I will have to get it done sooner rather than 
> later. I tried in the past to just upgrade Samba on its own - but ran 
> into a forest of issues with library version issues and compatibilities 
> - which broke other things on the server - so it doesn't seem like a 
> good idea.
> 
> Many thanks for all the hints
> 
>>
>>
>>>
>>> Some basic info:
>>> OS: Slackware 14.1
>>> Samba: 4.9.4
>>> Mode: Active Directory DC with file server on the same machine -
only
>>> one DC on domain
>>>
>>> Briefly, the samba_dlz plugin seems to be loading, but the logs
have
>>> various errors which so far I can't make sense of:
>>>
>>> Jun? 1 22:36:05 srv-01-op samba[11769]: 
>>> ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc - 
>>> NT_STATUS_ACCESS_DENIED
>>>
>>> and:
>>>
>>> # samba-tool dns zonelist localhost -U Administrator
>>> Password for [redacted\Administrator]:
>>> ERROR(runtime): uncaught exception - (9717, 
>>> 'WERR_DNS_ERROR_DS_UNAVAILABLE')
>>> ?? File 
>>>
"/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py", line
>>> 177, in _run
>>> ???? return self.run(*args, **kwargs)
>>> ?? File
"/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py",
>>> line 670, in run
>>> ???? request_filter)
>>>
>>> Should I just cut my loses and rebuild everything from scratch? It 
>>> will involve work from my part and downtime for the users, but I 
>>> should have really known about proper Samba AD db backups, so it is
>>> what it is.
>>>
>>> Any pointers much appreciated.
>>>
>

A small progress update. After some poking around, I noticed that 
ldbsearch reports 0 records in the following two ldb files:

/var/lib/samba/private/sam.ldb.d/DC\=FORESTDNSZONES,DC\=ORIGINPROBATE,DC\=LAN.ldb
/var/lib/samba/private/sam.ldb.d/DC\=DOMAINDNSZONES,DC\=ORIGINPROBATE,DC\=LAN.ldb

Copying their couterparts from /var/lib/samba/bind-dns/dns/sam.ldb.d 
seems to have restored the dns side of things. The were empty int he 
backups, so possibly this issue has been around for a while. I am 
obviously guessing, and don't understand properly what is the role of 
the files in the two different locations. I still can't join any new 
workstation to the domain, with the error:

"The specified server cannot perform the requested operationg.



On 02/06/2022 12:16, Sebastian Arcus via samba wrote:
> Hi Denis,
> 
> On 02/06/2022 07:49, Denis CARDON wrote:
>> Hi Sebastian,
>>
>> Le 01/06/2022 ? 23:54, Sebastian Arcus via samba a ?crit?:
>>> To start with the end, until today I never realised that there are 
>>> specific procedures for backing up Samba AD databases - which is my
>>> bad. I've always backed up /var/lib/samba and /var/cache/samba,
>>> seeing as that's where Samba kept its stuff. Today I've
accidentally
>>> deleted /var/lib/samba, and tried to copy it back from the nightly 
>>> backups. Needless to say that it all went to pots, and dns is not 
>>> working properly any more, not matter what I try. I can provide
more
>>> details and logs, but first I wanted to ask if it is even worth the
>>> effort? Is my backup of /var/lib/samba basically useless to restore
>>> things to where they were before?
>>
>> like Andrew said, as for every database you should use a proper 
>> coherent backup for samba ldb db files. That said, from experience it 
>> seldom fails (backups happen in the night when not much happens).
> 
> That's why I am a little bit stumped about what is going on here. I 
> tried backups of Samba databases from different days, even from the 
> weekend when I know there should be no activity on the server. Somehow 
> it doesn't make sense that they are all corrupted in the same way.
> 
>>
>> Could you try first to switch back to internal dns (if you where using 
>> bind-dlz), it should remove much issue with hardlinks and all. And 
>> turn off you bind9 on that machine. Then do a dbcheck --cross-ncs.
> 
> That sounded promising. I followed the instructions from Samba wiki, 
> shutdown Bind and Samba, and ran the command - for some reason it 
> results in the same error:
> 
> # samba_upgradedns --dns-backend=SAMBA_INTERNAL
> Reading domain information
> DNS accounts already exist
> No zone file /var/lib/samba/bind-dns/dns/redacted.LAN.zone
> DNS records will be automatically created
> Traceback (most recent call last):
>  ? File "/usr/sbin/samba_upgradedns", line 348, in <module>
>  ??? ncname = msg[0]['nCName'][0]
> KeyError: 'No such element'
> 
> I'm not sure why it is looking in /var/lib/samba/bind-dns/dns/ for a 
> zone file - I checked another healthy server and there is no zone file 
> there.
> 
>>
>> If you do a ldbsearch on the ldb files, does it crash?
> 
> That all seems fine. I did ldbsearch on all ldb files in 
> /var/lib/samba/bind-dns/dns/sam.ldb.d/ - and they all displayed the 
> content without issues
> 
> I'm really starting to wonder if I'm not barking up the wrong tree
with
> all this, and maybe it has nothing to do with corrupted backups somehow?
> 
>>
>> And if it goes back alive, please upgrade, there has been tons of 
>> bugfixes since samba 4.9.
> 
> That is good advice. This server has lots of other software running 
> (Asterisk/exim/dovecot/spamassassin etc) - so the upgrade is a 
> significant effort - but I will have to get it done sooner rather than 
> later. I tried in the past to just upgrade Samba on its own - but ran 
> into a forest of issues with library version issues and compatibilities 
> - which broke other things on the server - so it doesn't seem like a 
> good idea.
> 
> Many thanks for all the hints
> 
>>
>>
>>>
>>> Some basic info:
>>> OS: Slackware 14.1
>>> Samba: 4.9.4
>>> Mode: Active Directory DC with file server on the same machine -
only
>>> one DC on domain
>>>
>>> Briefly, the samba_dlz plugin seems to be loading, but the logs
have
>>> various errors which so far I can't make sense of:
>>>
>>> Jun? 1 22:36:05 srv-01-op samba[11769]: 
>>> ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc - 
>>> NT_STATUS_ACCESS_DENIED
>>>
>>> and:
>>>
>>> # samba-tool dns zonelist localhost -U Administrator
>>> Password for [redacted\Administrator]:
>>> ERROR(runtime): uncaught exception - (9717, 
>>> 'WERR_DNS_ERROR_DS_UNAVAILABLE')
>>> ?? File 
>>>
"/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py", line
>>> 177, in _run
>>> ???? return self.run(*args, **kwargs)
>>> ?? File
"/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py",
>>> line 670, in run
>>> ???? request_filter)
>>>
>>> Should I just cut my loses and rebuild everything from scratch? It 
>>> will involve work from my part and downtime for the users, but I 
>>> should have really known about proper Samba AD db backups, so it is
>>> what it is.
>>>
>>> Any pointers much appreciated.
>>>
>