samba - Oct 2024 - Garbage collection of tombstones is failing due to missing objects

Hello everyone,

I have currently two DCs running 4.21.1. The first DC is an older server
which was upgraded over time while the second one was added later to the
cluster and then upgraded as well. The recycle bin is enabled (which might
likely be the cause of this issue).

A while ago I noticed the following error in the log messages:
"garbage_collect_tombstones_part:
../../source4/dsdb/kcc/garbage_collect_tombstones.c:102: Failed to remove
deleted object [...]"

The error was logged multiple times, one time for each object. Following
this I executed "samba-tool domain tombstones expunge -d10". While
this
resulted in the same error messages, I at least got a clue about the reason:

{ "timestamp": "2024-10-28T07:01:58.612621+0100",
    "type": "dsdbChange",
    "dsdbChange": {
      "version": {
      "major": 1,
      "minor": 0
    },
    "statusCode": 32,
    "status": "No such object",
    "operation": "Delete",
    "remoteAddress": null,
    "performedAsSystem": false,
    "userSid": "S-1-5-18",
    "dn":
"CN=[...]\\0ADEL:cd01e963-eecd-4bb5-afda-eaac5513a120,CN=Deleted
Objects,DC=[...]",
    "transactionId": "1be00222-448e-4d0a-86b1-1e905d82fa1b",
    "sessionId": "f635f124-fcfa-40ad-9048-c2729c7738d9"
}}

A manual search in the database shows that the objects are indeed not
present. In fact, no deleted object exists in the database at all, only the
container "Deleted Objects" itself is listed in the results (ldbsearch
-H
ldap://127.0.0.1 'isDeleted=true' --show-deleted -U[...]).
It seems as if an object simply vanishes during deletion and is not moved
to the container for deleted objects. At the same time some kind of
dangling link is created pointing to the non existent object.
The truly strange part is that this only happens on the older DC. When I
search for deleted objects on the newer DC all deleted objects not yet
garbage collected are present in the deleted objects container and there
are also no problems logged during garbage collection.

Does anyone have an idea how I can fix this issue?

Running "samba-tool dbcheck --cross-ncs" did not solve this issue; the
tool
reports no errors.
One additional thing I found out is that in "DN: CN=Recycle Bin
Feature,CN=Optional Features,CN=Directory Service,CN=Windows
NT,CN=Services,CN=Configuration,DC=[...]" the older DC is listed in the
attribute "msDS-EnabledFeatureBL" while the newer DC is missing.
I am unsure if adding the newer DC improves or worsens the situation.

Cheers!

On 30/10/24 01:50, P. Heinrich via samba wrote:
> Hello everyone,
> 
> I have currently two DCs running 4.21.1. The first DC is an older server
> which was upgraded over time while the second one was added later to the
> cluster and then upgraded as well. The recycle bin is enabled (which might
> likely be the cause of this issue).
You are likely right. As far as I know, the recycle bin is rarely used 
with Samba AD, and it may not really be in a finished state.
> A manual search in the database shows that the objects are indeed not
> present. In fact, no deleted object exists in the database at all, only the
> container "Deleted Objects" itself is listed in the results
(ldbsearch -H
> ldap://127.0.0.1 'isDeleted=true' --show-deleted -U[...]).
> It seems as if an object simply vanishes during deletion and is not moved
> to the container for deleted objects. At the same time some kind of
> dangling link is created pointing to the non existent object.
> The truly strange part is that this only happens on the older DC. When I
> search for deleted objects on the newer DC all deleted objects not yet
> garbage collected are present in the deleted objects container and there
> are also no problems logged during garbage collection.
> 
> Does anyone have an idea how I can fix this issue?
No, however...
> One additional thing I found out is that in "DN: CN=Recycle Bin
> Feature,CN=Optional Features,CN=Directory Service,CN=Windows
> NT,CN=Services,CN=Configuration,DC=[...]" the older DC is listed in
the
> attribute "msDS-EnabledFeatureBL" while the newer DC is missing.
> I am unsure if adding the newer DC improves or worsens the situation.
if the two DCs are doing different things upon deletion, then 
replicating to each other, it is easy to imagine they'd end up with 
broken links.

The last comment on this bug is relevant:

https://bugzilla.samba.org/show_bug.cgi?id=10403

My guess (not advice!) is that if you either

1. add a msDS-EnabledFeatureBL to the new DC pointing to the Recycle Bin 
Feature, or
2. demote the old DC and add a new one without recycle bin

you will end up with quieter logs and more consistency. In one case, 
you'd end up with a recycling bin, and in the other you'd end up on the 
well-trodden path.

cheers,
Douglas