samba - Oct 2024 - Garbage collection of tombstones is failing due to missing objects

Hello everyone,

I have currently two DCs running 4.21.1. The first DC is an older server
which was upgraded over time while the second one was added later to the
cluster and then upgraded as well. The recycle bin is enabled (which might
likely be the cause of this issue).

A while ago I noticed the following error in the log messages:
"garbage_collect_tombstones_part:
../../source4/dsdb/kcc/garbage_collect_tombstones.c:102: Failed to remove
deleted object [...]"

The error was logged multiple times, one time for each object. Following
this I executed "samba-tool domain tombstones expunge -d10". While
this
resulted in the same error messages, I at least got a clue about the reason:

{ "timestamp": "2024-10-28T07:01:58.612621+0100",
    "type": "dsdbChange",
    "dsdbChange": {
      "version": {
      "major": 1,
      "minor": 0
    },
    "statusCode": 32,
    "status": "No such object",
    "operation": "Delete",
    "remoteAddress": null,
    "performedAsSystem": false,
    "userSid": "S-1-5-18",
    "dn":
"CN=[...]\\0ADEL:cd01e963-eecd-4bb5-afda-eaac5513a120,CN=Deleted
Objects,DC=[...]",
    "transactionId": "1be00222-448e-4d0a-86b1-1e905d82fa1b",
    "sessionId": "f635f124-fcfa-40ad-9048-c2729c7738d9"
}}

A manual search in the database shows that the objects are indeed not
present. In fact, no deleted object exists in the database at all, only the
container "Deleted Objects" itself is listed in the results (ldbsearch
-H
ldap://127.0.0.1 'isDeleted=true' --show-deleted -U[...]).
It seems as if an object simply vanishes during deletion and is not moved
to the container for deleted objects. At the same time some kind of
dangling link is created pointing to the non existent object.
The truly strange part is that this only happens on the older DC. When I
search for deleted objects on the newer DC all deleted objects not yet
garbage collected are present in the deleted objects container and there
are also no problems logged during garbage collection.

Does anyone have an idea how I can fix this issue?

Running "samba-tool dbcheck --cross-ncs" did not solve this issue; the
tool
reports no errors.
One additional thing I found out is that in "DN: CN=Recycle Bin
Feature,CN=Optional Features,CN=Directory Service,CN=Windows
NT,CN=Services,CN=Configuration,DC=[...]" the older DC is listed in the
attribute "msDS-EnabledFeatureBL" while the newer DC is missing.
I am unsure if adding the newer DC improves or worsens the situation.

Cheers!