samba - Sep 2012 - Samba4 AD returning incomplete results, can't edit much, and unable to reindex sam.ldb

Not sure if this is best sent to samba or samba-technical. If an admin thinks
this is best sent to technical, let me know and I'll resend it there.

Hi all, 

Well I've managed to land myself in a bit of a heap. 

I've been successfully running Samba4 at two schools I administer as well as
my home test network, starting with alpha17 and now on beta8. Group Policy works
a charm.

But lately at one of the schools it seems sam.ldb has got messed up. 

In ADUC, I can browse all users, groups and machines fine, everything and
everyone authenticates and operates fine, and I can view all current group
memberships for a user and existing users in a group, but if searching for users
to add to a group, or to add groups to a user, only a small subset of
users/groups (respectively) shows in the search results, and if I type the name
of any other user/group manually, I get told by ADUC that they do not exist. I
only see 6 out of about 90 users, and 22 out of about 65 groups (all the
builtins seem to show as part of that 22).

This means my AD is more or less stuck from an administrative point of view. I
can generally not change user or group memberships without difficulty.
This looks like it happened while I was on leave for a few weeks, so backups of
non-corrupt data have been overwritten - I only had a two-week
rotation/retention policy on /srv/adsrv/var contents (changed since!).

So in ADUC I can view group members or view user groups but can't modify the
bulk of them.

samba-tool behaves the opposite - "samba-tool group listmembers
(groupname)" only lists users if they're in the same set of 6, but
addmember succeeds - if I use addmember, while listmembers still doesn't
show the newly added member to a group, opening the group in ADUC does list the
member.

I can't discern any pattern or common element exclusive to those 6 users. 

If I do a 'ldbsearch -H sam.ldb "objectClass=*"', out of the
user records returned, only the same 6 that show up in AD searches show up in
the results (amongst other machine and non-user objects).

samba-tool dbcheck --cross-ncs returns "Checked 3229 objects (0
errors)", but samba-tool dbcheck --reindex fails with:

=========================== 
Re-indexing... 
Invalid data for index
DC=_kerberos._tcp.Default-First-Site-Name._sites.dc,DC=_msdcs.ad.(domain
name),CN=MicrosoftDNS,DC=ForestDnsZones,DC=ad,DC=(domain name)

ltdb: tdb(/srv/adsrv/var/lib/samba/private/sam.ldb.d/DC=AD,DC=(domain
name).ldb): tdb_rec_read bad magic 0x6863733d at offset=1773572

re-indexed database : (1, "attribute 'force_reindex': no matching
attribute value while deleting attribute on '@ATTRIBUTES'")
=========================== 

(I have the samba4 tree contained in /srv/adsrv on this server to isolate it
from a samba 3 instance doing the file sharing, inspired by "Franky" -
this is left over from a configuration to suit alpha17 (the smbd subprocess
didn't work back then for shares) and otherwise works fine, also works fine
at the other school).

I can't browse past the
Default-First-Site-Name._sites.dc,DC=_msdcs.ad.(domain
name),CN=MicrosoftDNS,DC=ForestDnsZones,DC=ad,DC=(domain name) folder using the
Windows-based LDAP_Admin.exe utility, it throws this error:
"LDAP error! Operations Error: 00002020: schema: metadata tdb not
initialized at ../source4/dsdb/samdb/ldb_modules/schema_load.c:117"

Based on the advice here: 

http://lists.samba.org/archive/samba-technical/2010-December/075239.html 

... I tried to manually remove the index by doing this: 

/srv/adsrv/bin/ldbedit -H /srv/adsrv/var/lib/samba/private/sam.ldb -s base -b
\@INDEXLIST

... and clearing out the index to the example given in the above link. Or even
just removing one entry. However, any modifications fail with a similar error to
the above reindex command:

=========================== 
ltdb: tdb(/srv/adsrv/var/lib/samba/private/sam.ldb.d/DC=AD,DC(domain name).ldb):
tdb_rec_read bad magic 0x6863733d at offset=1773572

failed to modify @INDEXLIST - ldb_wait: Operations error (1) 
=========================== 
... and the modification doesn't happen. 

Argh! 

Any ides as to how I may be able to get out of this? Any help appreciated. 

Regards, 


Andrew