samba - Jun 2022 - Restoring Samba databases from /var/lib/samba

To start with the end, until today I never realised that there are 
specific procedures for backing up Samba AD databases - which is my bad. 
I've always backed up /var/lib/samba and /var/cache/samba, seeing as 
that's where Samba kept its stuff. Today I've accidentally deleted 
/var/lib/samba, and tried to copy it back from the nightly backups. 
Needless to say that it all went to pots, and dns is not working 
properly any more, not matter what I try. I can provide more details and 
logs, but first I wanted to ask if it is even worth the effort? Is my 
backup of /var/lib/samba basically useless to restore things to where 
they were before?

Some basic info:
OS: Slackware 14.1
Samba: 4.9.4
Mode: Active Directory DC with file server on the same machine - only 
one DC on domain

Briefly, the samba_dlz plugin seems to be loading, but the logs have 
various errors which so far I can't make sense of:

Jun  1 22:36:05 srv-01-op samba[11769]: 
../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc - 
NT_STATUS_ACCESS_DENIED

and:

# samba-tool dns zonelist localhost -U Administrator
Password for [redacted\Administrator]:
ERROR(runtime): uncaught exception - (9717,
'WERR_DNS_ERROR_DS_UNAVAILABLE')
   File "/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py",
line 177, in _run
     return self.run(*args, **kwargs)
   File "/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py", line
670, in run
     request_filter)

Should I just cut my loses and rebuild everything from scratch? It will 
involve work from my part and downtime for the users, but I should have 
really known about proper Samba AD db backups, so it is what it is.

Any pointers much appreciated.

On Wed, 2022-06-01 at 22:54 +0100, Sebastian Arcus via samba
wrote:
> To start with the end, until today I never realised that there are 
> specific procedures for backing up Samba AD databases - which is my
> bad. 
> I've always backed up /var/lib/samba and /var/cache/samba, seeing as 
> that's where Samba kept its stuff. Today I've accidentally deleted 
> /var/lib/samba, and tried to copy it back from the nightly backups. 
> Needless to say that it all went to pots, and dns is not working 
> properly any more, not matter what I try. I can provide more details
> and 
> logs, but first I wanted to ask if it is even worth the effort? Is
> my 
> backup of /var/lib/samba basically useless to restore things to
> where 
> they were before?
samba_upgradedns can fix the links for BIND9_DLZ, but your issues seem
worse than that. 
> Some basic info:
> OS: Slackware 14.1
> Samba: 4.9.4
> Mode: Active Directory DC with file server on the same machine -
> only 
> one DC on domain
> 
> Briefly, the samba_dlz plugin seems to be loading, but the logs have 
> various errors which so far I can't make sense of:
> 
> Jun  1 22:36:05 srv-01-op samba[11769]: 
> ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc - 
> NT_STATUS_ACCESS_DENIED
> 
> and:
> 
> # samba-tool dns zonelist localhost -U Administrator
> Password for [redacted\Administrator]:
> ERROR(runtime): uncaught exception - (9717,
> 'WERR_DNS_ERROR_DS_UNAVAILABLE')
>    File "/usr/lib64/python2.7/site-
> packages/samba/netcmd/__init__.py", 
> line 177, in _run
>      return self.run(*args, **kwargs)
>    File "/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py",
> line 
> 670, in run
>      request_filter)
I would look at the server logs more, and things like 'samba-tool
dbcheck --cross-ncs'
> Should I just cut my loses and rebuild everything from scratch? It
> will 
> involve work from my part and downtime for the users, but I should
> have 
> really known about proper Samba AD db backups, so it is what it is.
> 
> Any pointers much appreciated.
Our DBs need to be backed up with the locks taken, otherwise you can
find it mid-modify.  Otherwise it is just pure luck as to if it was
quiet at the time. 

Might be worth engaging some professional help.

Andrew Bartlett

-- 
Andrew Bartlett (he/him)       https://samba.org/~abartlet/
Samba Team Member (since 2001) https://samba.org
Samba Team Lead, Catalyst IT   https://catalyst.net.nz/services/samba

Samba Development and Support, Catalyst IT - Expert Open Source
Solutions

Hi Sebastian,

Le 01/06/2022 ? 23:54, Sebastian Arcus via samba a
?crit?:
> To start with the end, until today I never realised that there are 
> specific procedures for backing up Samba AD databases - which is my bad. 
> I've always backed up /var/lib/samba and /var/cache/samba, seeing as 
> that's where Samba kept its stuff. Today I've accidentally deleted 
> /var/lib/samba, and tried to copy it back from the nightly backups. 
> Needless to say that it all went to pots, and dns is not working 
> properly any more, not matter what I try. I can provide more details and 
> logs, but first I wanted to ask if it is even worth the effort? Is my 
> backup of /var/lib/samba basically useless to restore things to where 
> they were before?
like Andrew said, as for every database you should use a proper coherent 
backup for samba ldb db files. That said, from experience it seldom 
fails (backups happen in the night when not much happens).

Could you try first to switch back to internal dns (if you where using 
bind-dlz), it should remove much issue with hardlinks and all. And turn 
off you bind9 on that machine. Then do a dbcheck --cross-ncs.

If you do a ldbsearch on the ldb files, does it crash?

And if it goes back alive, please upgrade, there has been tons of 
bugfixes since samba 4.9.

Cheers,

Denis

> 
> Some basic info:
> OS: Slackware 14.1
> Samba: 4.9.4
> Mode: Active Directory DC with file server on the same machine - only 
> one DC on domain
> 
> Briefly, the samba_dlz plugin seems to be loading, but the logs have 
> various errors which so far I can't make sense of:
> 
> Jun? 1 22:36:05 srv-01-op samba[11769]: 
> ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc - 
> NT_STATUS_ACCESS_DENIED
> 
> and:
> 
> # samba-tool dns zonelist localhost -U Administrator
> Password for [redacted\Administrator]:
> ERROR(runtime): uncaught exception - (9717, 
> 'WERR_DNS_ERROR_DS_UNAVAILABLE')
>  ? File
"/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py",
> line 177, in _run
>  ??? return self.run(*args, **kwargs)
>  ? File "/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py",
line
> 670, in run
>  ??? request_filter)
> 
> Should I just cut my loses and rebuild everything from scratch? It will 
> involve work from my part and downtime for the users, but I should have 
> really known about proper Samba AD db backups, so it is what it is.
> 
> Any pointers much appreciated.
>

On 6/1/22 5:54 PM, Sebastian Arcus via samba wrote:
> To start with the end, until today I never realised that there are 
> specific procedures for backing up Samba AD databases - which is my bad. 
> I've always backed up /var/lib/samba and /var/cache/samba, seeing as 
> that's where Samba kept its stuff. Today I've accidentally deleted 
> /var/lib/samba, and tried to copy it back from the nightly backups. 
> Needless to say that it all went to pots, and dns is not working 
> properly any more, not matter what I try. I can provide more details and 
> logs, but first I wanted to ask if it is even worth the effort? Is my 
> backup of /var/lib/samba basically useless to restore things to where 
> they were before?
> 
I have never had a problem moving Samba DCs from one node to another, 
with file copying, without using Samba backup features or demoting and 
adding a new DC. I run them as containers, so all the state in 
/var/lib/samba is properly isolated from the rest of the system.

What you must take into account is:

1) Backup ACLs and entire list of Extended Attributes, specially the 
Samba specific ones (this is for SYSVOL permissions)

2) Use a filesystem snapshot so the state is backed up intact, If the Dc 
is running at the time of the backup it will be restored as a crashed 
instance. You can stop, make the snapshot start inmediatly and then make 
the backup of the snapshot.

3) Or you can stop it and make a backup without a snaphot and later 
start it.

So your problem maybe complex because you may have database 
inconsistencies caused by how the backup was made lik,e files being 
modified while being bnacked up.

> Some basic info:
> OS: Slackware 14.1
> Samba: 4.9.4
> Mode: Active Directory DC with file server on the same machine - only 
> one DC on domain
> 
> Briefly, the samba_dlz plugin seems to be loading, but the logs have 
> various errors which so far I can't make sense of:
> 
> Jun? 1 22:36:05 srv-01-op samba[11769]: 
> ../source4/dsdb/kcc/kcc_periodic.c:768: Failed samba_kcc - 
> NT_STATUS_ACCESS_DENIED
> 
> and:
> 
> # samba-tool dns zonelist localhost -U Administrator
> Password for [redacted\Administrator]:
> ERROR(runtime): uncaught exception - (9717, 
> 'WERR_DNS_ERROR_DS_UNAVAILABLE')
>  ? File
"/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py",
> line 177, in _run
>  ??? return self.run(*args, **kwargs)
>  ? File "/usr/lib64/python2.7/site-packages/samba/netcmd/dns.py",
line
> 670, in run
>  ??? request_filter)
> 
> Should I just cut my loses and rebuild everything from scratch? It will 
> involve work from my part and downtime for the users, but I should have 
> really known about proper Samba AD db backups, so it is what it is.
> 
> Any pointers much appreciated.
>