samba - Jan 2016 - Samba 4 AD - Samba Fails to Start, hdb_samba4_create_kdc (setup KDC database) failed

Andrew Bartlett <abartlet <at> samba.org> writes:

 
> 
> What does 'samba-tool dbcheck' say?  
Running "sudo samba-tool dbfix" produces the following Python error:

sudo samba-tool dbcheck
ERROR(<type 'exceptions.IndexError'>): uncaught exception - list
index out
of range
  File "/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py",
line
175, in _run
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/samba/netcmd/dbcheck.py",
line 120,
in run
    reset_well_known_acls=reset_well_known_acls)
  File "/usr/lib/python2.7/dist-packages/samba/dbchecker.py", line 87,
in
__init__
    dnsadmins_sid = ndr_unpack(security.dom_sid,
res[0]["objectSid"][0])

Appreciate you joining the conversation Andrew, do you think CrashPlan
corrupted this database?  I can't think of anything else I could have done
that would've caused such a drastic failure and would like to know so I
don't repeat the blunder in the future, this has been a royal PITA.

JS

On Sun, 2016-01-03 at 10:31 +0000, JS wrote:
> Andrew Bartlett <abartlet <at> samba.org> writes:
> 
>  
> > 
> > What does 'samba-tool dbcheck' say?  
> 
> Running "sudo samba-tool dbfix" produces the following Python
error:
> 
> sudo samba-tool dbcheck
> ERROR(<type 'exceptions.IndexError'>): uncaught exception -
list
> index out
> of range
>   File
"/usr/lib/python2.7/dist-packages/samba/netcmd/__init__.py",
> line
> 175, in _run
>     return self.run(*args, **kwargs)
>   File
"/usr/lib/python2.7/dist-packages/samba/netcmd/dbcheck.py",
> line 120,
> in run
>     reset_well_known_acls=reset_well_known_acls)
>   File "/usr/lib/python2.7/dist-packages/samba/dbchecker.py",
line
> 87, in
> __init__
>     dnsadmins_sid = ndr_unpack(security.dom_sid,
> res[0]["objectSid"][0])
> 
> Appreciate you joining the conversation Andrew, do you think
> CrashPlan
> corrupted this database?  I can't think of anything else I could have
> done
> that would've caused such a drastic failure and would like to know so
> I
> don't repeat the blunder in the future, this has been a royal PITA.
Is there really a Samba database in the location shown by:

bin/testparm --parameter-name=privatedir --suppress-prompt

That is, a sam.ldb, a secrets.ldb and (importantly) sam.ldb.d/ with the
usual files in that (metadata.tdb, other files ending in .ldb named
after your domain).

It looks to me like this has been removed (or we have been pointed at
the wrong location), and Samba has re-created an empty DB for sam.ldb,
with nothing in it.  I mention this because the alternative is that it
is damaged beyond (costly/tedious/manual) repair involving a rebuild
and putting back some of the old values. 

The last time I came across a DB failure like this, I blamed a DRDB
setup that didn't honour 'barriers' and an unexpected power-off. 
The
DB was only able to be partially rescued with the new 'ldbdump' tool we
wrote.  In that case the domain was able to hobble on for a few weeks,
but was rebuilt.

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

Andrew Bartlett <abartlet <at> samba.org> writes:
> Is there really a Samba database in the location shown by:
> 
> bin/testparm --parameter-name=privatedir --suppress-prompt
> 
> That is, a sam.ldb, a secrets.ldb and (importantly) sam.ldb.d/ with the
> usual files in that (metadata.tdb, other files ending in .ldb named
> after your domain).

Hi Andrew,

Here is the result of your testparm command:

sudo testparm --parameter-name=privatedir --suppress-prompt
 
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Processing section "[netlogon]"
Processing section "[sysvol]"
Processing section "[accounting]"
Processing section "[data]"
Processing section "[backups]"
Loaded services file OK.
/var/lib/samba/private



ls -la /var/lib/samba/private/
total 11220
drwxr-xr-x 6 root root    4096 Dec 28 21:12 .
drwxr-xr-x 8 root root    4096 Dec 13 21:07 ..
-rw------- 1 root root    2085 Dec 13 21:07 dns_update_cache
-rw-r--r-- 1 root root    3183 Dec 13 21:03 dns_update_list
-rw------- 1 root root 1286144 Dec 13 21:02 hklm.ldb
-rw------- 1 root root 1609728 Dec 23 20:15 idmap.ldb
-rw-r--r-- 1 root root      99 Dec 13 21:03 krb5.conf
srwxrwxrwx 1 root root       0 Dec 28 21:12 ldapi
drwxr-x--- 2 root root    4096 Dec 28 21:12 ldap_priv
-r--r--r-- 1 root root     242 Dec 13 21:07 named.conf.update
-rw------- 1 root root 1286144 Dec 13 21:41 privilege.ldb
-rw------- 1 root root     696 Dec 13 21:07 randseed.tdb
-rw------- 1 root root 4247552 Dec 28 07:22 sam.ldb
drwx------ 2 root root    4096 Dec 13 21:02 sam.ldb.d
-rw------- 1 root root     696 Dec 28 21:12 schannel_store.tdb
-rw------- 1 root root    1212 Dec 13 21:03 secrets.keytab
-rw------- 1 root root 1286144 Dec 13 21:03 secrets.ldb
-rw------- 1 root root  430080 Dec 13 21:03 secrets.tdb
-rw------- 1 root root 1286144 Dec 13 21:02 share.ldb
drwxr-xr-x 3 root root    4096 Dec 13 21:07 smbd.tmp
-rw-r--r-- 1 root root     955 Dec 13 21:03 spn_update_list
drwx------ 2 root root    4096 Dec 13 21:07 tls



sudo ls -la /var/lib/samba/private/sam.ldb.d/
total 39000
drwx------ 2 root root     4096 Dec 13 21:02 .
drwxr-xr-x 6 root root     4096 Dec 28 21:12 ..
-rw------- 1 root root 16384000 Dec 28 07:22
CN=CONFIGURATION,DC=ONE,DC=CLIFFBELLS,DC=COM.ldb
-rw------- 1 root root 10383360 Dec 28 07:22
CN=SCHEMA,CN=CONFIGURATION,DC=ONE,DC=CLIFFBELLS,DC=COM.ldb
-rw------- 1 root root  4247552 Dec 28 07:22
DC=DOMAINDNSZONES,DC=ONE,DC=CLIFFBELLS,DC=COM.ldb
-rw------- 1 root root  4247552 Dec 28 07:22
DC=FORESTDNSZONES,DC=ONE,DC=CLIFFBELLS,DC=COM.ldb
-rw------- 1 root root  4243456 Dec 28 07:22 DC=ONE,DC=CLIFFBELLS,DC=COM.ldb
-rw-r----- 1 root root   421888 Dec 27 21:44 metadata.tdb
 
> 
> It looks to me like this has been removed (or we have been pointed at
> the wrong location), and Samba has re-created an empty DB for sam.ldb,
> with nothing in it.  I mention this because the alternative is that it
> is damaged beyond (costly/tedious/manual) repair involving a rebuild
> and putting back some of the old values. 
It looks to me like everything is correct there... 
> 
> The last time I came across a DB failure like this, I blamed a DRDB
> setup that didn't honour 'barriers' and an unexpected
power-off.  The
> DB was only able to be partially rescued with the new 'ldbdump'
tool we
> wrote.  In that case the domain was able to hobble on for a few weeks,
> but was rebuilt.
> 
> Andrew Bartlett
> 
Not sre if the ldbdump tool you mention could help in this scenario or not.
 This machine is on an APC UPS so sudden shutdown shouldn't have been an
issue.

JS