samba - Jan 2018 - [Patches] AD Database corruption after upgrade from <= 4.6 to 4.7 (bug #13228)

Hi,

here're patches to avoid a database corruption with linked attributes,
e.g. member/memberOf.

See https://bugzilla.samba.org/show_bug.cgi?id=13228

As a temporary solution admins can add "server services = -kcc" to the
global section of smb.conf.

Also DO NOT repair the following errors with samba-tool dbcheck!
"Remove duplicate links in attribute"
and
"ERROR: orphaned backlink"
as this removes the ability to repair the database
in the next round of patches!

Please review and push:-)

Thanks!
metze
-------------- next part --------------

On Mon, Jan 22, 2018 at 10:49:29AM +0100, Stefan Metzmacher via samba-technical
wrote:
> Please review and push:-)
lgtm&pushed.

-slow

-- 
Ralph Boehme, Samba Team       https://samba.org/
Samba Developer, SerNet GmbH   https://sernet.de/en/samba/

On Mon, 2018-01-22 at 11:34 +0100, Ralph Böhme via samba
wrote:
> On Mon, Jan 22, 2018 at 10:49:29AM +0100, Stefan Metzmacher via
samba-technical wrote:
> > Please review and push:-)
> 
> lgtm&pushed.
I'm also happy with them (but it is late here and so I didn't want to
give that as a formal review till I woke up tomorrow :-).

It took a moment to get why the 'unsorted' links were sorted, at this
time of night every GUID looks the same, but they are subtly altered to
cause a specific re-sort order. 

Thank you very much!

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

Am 22.01.2018 um 10:49 schrieb Stefan Metzmacher via
samba:
> Also DO NOT repair the following errors with samba-tool dbcheck!
> "Remove duplicate links in attribute"
> and
> "ERROR: orphaned backlink"
> as this removes the ability to repair the database
> in the next round of patches!
>
I had this error after upgrading from 4.7.3 to 4.7.4 and used samba-tool 
dbcheck --clean to get rid of them.
Replication is still working. What kind of unrepairable corruption can i 
expect now?

Thanks in advance,
Achim~

On Mon, Jan 22, 2018 at 05:24:44PM +0100, Achim Gottinger via samba
wrote:
> Am 22.01.2018 um 10:49 schrieb Stefan Metzmacher via samba:
> > Also DO NOT repair the following errors with samba-tool dbcheck!
> > "Remove duplicate links in attribute"
> > and
> > "ERROR: orphaned backlink"
> > as this removes the ability to repair the database
> > in the next round of patches!
> > 
> I had this error after upgrading from 4.7.3 to 4.7.4 and used samba-tool
> dbcheck --clean to get rid of them.
> Replication is still working. What kind of unrepairable corruption can i
> expect now?
see the bug report for details, this can eg cause loss of group memberships or
generally speaking loss of linked-attributes.

The only remede is comparing all objects for differences in linked-attributes
and restore overwritten forward-links from now dangling backlinks.

We're currently also working on an improvement to dbcheck so it can detect
such
corruption and fix it, but this will only work if you did *not* run dbcheck
--fix on the affected database.

-slow

-- 
Ralph Boehme, Samba Team       https://samba.org/
Samba Developer, SerNet GmbH   https://sernet.de/en/samba/

Hi,

as a lot of SerNet customers are having trouble with corrupted
linked attributes, my colleague Ralph Böhme and I developed
patches for 'samba-tool dbcheck' to recover the missing
forward links (in most cases missing member attributes).

I'm currently running a private autobuild with these patches
and my colleague Björn Baumbach is currently testing SAMBA+
packages with the patches included, which will be released
as soon as possible.

As the patches re-add members to groups administrators may want
avoid using '--yes' and ack the re-added members explicitly.

The patches have enough review tags already, additional
review isn't required, we'll wait a bit to collect some feedback
from others, before pushing.

Once the patches are reviewed for master, we'll also release
a new upstream 4.7 release with the fixes included.

More technical details:

As we lost the replication meta data for the forward link,
we create them using a special invocationId
ffffffff-4700-4700-4700-000000b13228 and an originating_usn
of 1. The add/changetime/local_usn are the one from the last
'objectClass' modification (which typically never changes and therefor
matches the object creation time). We also use version = 0
in order to match the link creation of 4.7 and older releases.

This way we can easily identify recreated forward links
and we avoid a new meta data stamp and incrementing of
the highestCommitedUSN. So each affected dc will just recover
the value in the local database. And any incoming
replication should overwrite the value again.

See also https://bugzilla.samba.org/show_bug.cgi?id=13228

metze

Am 22.01.2018 um 10:49 schrieb Stefan Metzmacher via
samba-technical:
> Hi,
> 
> here're patches to avoid a database corruption with linked attributes,
> e.g. member/memberOf.
> 
> See https://bugzilla.samba.org/show_bug.cgi?id=13228
> 
> As a temporary solution admins can add "server services = -kcc"
to the
> global section of smb.conf.
> 
> Also DO NOT repair the following errors with samba-tool dbcheck!
> "Remove duplicate links in attribute"
> and
> "ERROR: orphaned backlink"
> as this removes the ability to repair the database
> in the next round of patches!
> 
> Please review and push:-)
> 
> Thanks!
> metze
> 

-------------- next part --------------

Hi Stefan

I am also one of the Sernet customer. Can you guide me how to run the patch
to fix the bug.

I am running 2 DC's Sernet Samba 4.7.4 with 2 RODC's running Sernet
Samba
4.7.4. Whenever I run samba-tool drs replicate --fix --yes command on the
DC it shows the below errors which cannot be fixed:

Failed to remove deleted DN attribute fromServer : (65, "objectclass_attrs:
at least one mandatory attribute ('fromServer') on entry
'CN=79fbbaa2-a6b5-4dfd-a7f4-26aaa568f74e,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na'
wasn't specified!")
ERROR: no target object found for GUID component for link lastKnownParent
in object
CN=79fbbaa2-a6b5-4dfd-a7f4-26aaa568f74e,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na
- <GUID=fbd5524d-78fb-4313-a62a-96dc802dd9e2>;CN=NTDS
Settings\0ADEL:fbd5524d-78fb-4313-a62a-96dc802dd9e2,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na
ERROR: target DN is deleted for lastKnownParent in object
CN=79fbbaa2-a6b5-4dfd-a7f4-26aaa568f74e,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na
- <GUID=fbd5524d-78fb-4313-a62a-96dc802dd9e2>;CN=NTDS
Settings\0ADEL:fbd5524d-78fb-4313-a62a-96dc802dd9e2,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na
Target GUID points at deleted DN
'<GUID=fbd5524d-78fb-4313-a62a-96dc802dd9e2>;CN=NTDS
Settings\\0ADEL:fbd5524d-78fb-4313-a62a-96dc802dd9e2,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na'
Remove DN link? [YES]
Failed to remove deleted DN attribute lastKnownParent : (65,
"objectclass_attrs: at least one mandatory attribute ('fromServer')
on
entry
'CN=79fbbaa2-a6b5-4dfd-a7f4-26aaa568f74e,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na'
wasn't specified!")
WARNING: no target object found for GUID component for DN value fromServer
in object
CN=6eba8ddc-5f5b-4bf5-8025-772ec80a29e2,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na
- <GUID=3da7e1da-33b5-428b-9313-2ae48ddfee10>;CN=NTDS
Settings,CN=IUMONGDC,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=iumnet,DC=edu,DC=na
WARNING: target DN is deleted for fromServer in object
CN=6eba8ddc-5f5b-4bf5-8025-772ec80a29e2,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na
- <GUID=3da7e1da-33b5-428b-9313-2ae48ddfee10>;CN=NTDS
Settings,CN=IUMONGDC,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=iumnet,DC=edu,DC=na
Target GUID points at deleted DN
'<GUID=3da7e1da-33b5-428b-9313-2ae48ddfee10>;CN=NTDS
Settings,CN=IUMONGDC,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=iumnet,DC=edu,DC=na'
Remove stale DN link? [YES]
Failed to remove deleted DN attribute fromServer : (65, "objectclass_attrs:
at least one mandatory attribute ('fromServer') on entry
'CN=6eba8ddc-5f5b-4bf5-8025-772ec80a29e2,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na'
wasn't specified!")
ERROR: no target object found for GUID component for link lastKnownParent
in object
CN=6eba8ddc-5f5b-4bf5-8025-772ec80a29e2,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na
- <GUID=fbd5524d-78fb-4313-a62a-96dc802dd9e2>;CN=NTDS
Settings\0ADEL:fbd5524d-78fb-4313-a62a-96dc802dd9e2,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na
ERROR: target DN is deleted for lastKnownParent in object
CN=6eba8ddc-5f5b-4bf5-8025-772ec80a29e2,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na
- <GUID=fbd5524d-78fb-4313-a62a-96dc802dd9e2>;CN=NTDS
Settings\0ADEL:fbd5524d-78fb-4313-a62a-96dc802dd9e2,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na
Target GUID points at deleted DN
'<GUID=fbd5524d-78fb-4313-a62a-96dc802dd9e2>;CN=NTDS
Settings\\0ADEL:fbd5524d-78fb-4313-a62a-96dc802dd9e2,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na'
Remove DN link? [YES]
Failed to remove deleted DN attribute lastKnownParent : (65,
"objectclass_attrs: at least one mandatory attribute ('fromServer')
on
entry
'CN=6eba8ddc-5f5b-4bf5-8025-772ec80a29e2,CN=LostAndFoundConfig,CN=Configuration,DC=iumnet,DC=edu,DC=na'
wasn't specified!")
Checked 5920 objects (13 errors)

Can you please suggest if this patch is going to fix these errors.

Thanks n Regards

Harsh

*Harsh Kukreja *Systems Administrator
*International University of Namibia *Tel: 061-4336000 - E-mail: h.kukreja
@ium.edu.na - Web:
*http://www.ium.edu.na <http://www.ium.edu.na/>*Private Bag
14005,Bachbrech. 21-31 Hercules Street, Dorado Park, Windhoek, NAMIBIA






On Tue, Jan 30, 2018 at 8:56 PM, Stefan Metzmacher via samba <
samba at lists.samba.org> wrote:
> Hi,
>
> as a lot of SerNet customers are having trouble with corrupted
> linked attributes, my colleague Ralph Böhme and I developed
> patches for 'samba-tool dbcheck' to recover the missing
> forward links (in most cases missing member attributes).
>
> I'm currently running a private autobuild with these patches
> and my colleague Björn Baumbach is currently testing SAMBA+
> packages with the patches included, which will be released
> as soon as possible.
>
> As the patches re-add members to groups administrators may want
> avoid using '--yes' and ack the re-added members explicitly.
>
> The patches have enough review tags already, additional
> review isn't required, we'll wait a bit to collect some feedback
> from others, before pushing.
>
> Once the patches are reviewed for master, we'll also release
> a new upstream 4.7 release with the fixes included.
>
> More technical details:
>
> As we lost the replication meta data for the forward link,
> we create them using a special invocationId
> ffffffff-4700-4700-4700-000000b13228 and an originating_usn
> of 1. The add/changetime/local_usn are the one from the last
> 'objectClass' modification (which typically never changes and
therefor
> matches the object creation time). We also use version = 0
> in order to match the link creation of 4.7 and older releases.
>
> This way we can easily identify recreated forward links
> and we avoid a new meta data stamp and incrementing of
> the highestCommitedUSN. So each affected dc will just recover
> the value in the local database. And any incoming
> replication should overwrite the value again.
>
> See also https://bugzilla.samba.org/show_bug.cgi?id=13228
>
> metze
>
> Am 22.01.2018 um 10:49 schrieb Stefan Metzmacher via samba-technical:
> > Hi,
> >
> > here're patches to avoid a database corruption with linked
attributes,
> > e.g. member/memberOf.
> >
> > See https://bugzilla.samba.org/show_bug.cgi?id=13228
> >
> > As a temporary solution admins can add "server services =
-kcc" to the
> > global section of smb.conf.
> >
> > Also DO NOT repair the following errors with samba-tool dbcheck!
> > "Remove duplicate links in attribute"
> > and
> > "ERROR: orphaned backlink"
> > as this removes the ability to repair the database
> > in the next round of patches!
> >
> > Please review and push:-)
> >
> > Thanks!
> > metze
> >
>
>
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>