samba - Jun 2017 - skip chunk if "DRS linked attribute for GUID - DN not found"

Hello everyone,

i try to use Samba RODC(4.6.5) with W2K8R2. Windows AD has around 35000 objects.
My Samba machine is small one (ARM 32bit CPU) with only 2GB physical memory, so
i can’t join to the domain because of expensive memory usage.
To solve this Problem, i decide to replicate only critical objects and then let
samba_kcc to get other objects.

1 ) Is this an possible way to use Samba AD or should i replicate all objects
while joining new ad?

As result i got fast join and no more problems with memory usage.
But i got some other errors for example:

../source4/dsdb/repl/replicated_objects.c:933 Failed to prepare commit of
transaction: DRS linked attribute for GUID 6709d915-7dfb-41bf-b695-dd1a798d7718
- DN not found

Object with this objectGUID exists in AD, but was not replicated yet. So i
create to patches follow to skip the chunks with „problem“-objects and try to
get the other missing objects with effect, that links can be created and i can
replicate rest.

2) Is it rigth way to raise replication on such errors? Maybe it is better to
skip the chunk and replicate other stuff first and let samba_kcc do his work and
repeat the skiped chunks?

Thanks,

Andrej

(patches should show the idea)

diff --git a/source4/dsdb/repl/replicated_objects.c
b/source4/dsdb/repl/replicated_objects.c
index 637f7fa..a103e55 100644
--- a/source4/dsdb/repl/replicated_objects.c
+++ b/source4/dsdb/repl/replicated_objects.c
@@ -922,6 +922,18 @@ WERROR dsdb_replicated_objects_commit(struct ldb_context
*ldb,
                }

               ret = ldb_transaction_prepare_commit(ldb);
+        if (ret == LDB_ERR_NO_SUCH_OBJECT) {
+                             /* restore previous schema */
+                             if (used_global_schema) {
+                                             dsdb_set_global_schema(ldb);
+                             } else if (cur_schema ) {
+                                             dsdb_reference_schema(ldb,
cur_schema, false);
+                             }
+                             DEBUG(0,(__location__ " Failed to prepare
commit of transaction: %s\n",
+                                             ldb_errstring(ldb)));
+                             TALLOC_FREE(tmp_ctx);
+                             return WERR_DS_NO_SUCH_OBJECT;
+             }
                if (ret != LDB_SUCCESS) {
                               /* restore previous schema */
                               if (used_global_schema) {


diff --git a/source4/dsdb/repl/drepl_out_helpers.c
b/source4/dsdb/repl/drepl_out_helpers.c
index ac0b947..68774f4 100644
--- a/source4/dsdb/repl/drepl_out_helpers.c
+++ b/source4/dsdb/repl/drepl_out_helpers.c
@@ -916,7 +916,7 @@ static void
dreplsrv_op_pull_source_apply_changes_trigger(struct tevent_req *req
                                                                                
&state->op->source_dsa->notify_uSN);
               talloc_free(objects);
-              if (!W_ERROR_IS_OK(status)) {
+             if (!W_ERROR_IS_OK(status) && !W_ERROR_EQUAL(status,
WERR_DS_NO_SUCH_OBJECT)) {
                               /*
                               * If we failed to apply the records due to a
missing
@@ -943,6 +943,11 @@ static void
dreplsrv_op_pull_source_apply_changes_trigger(struct tevent_req *req
                              tevent_req_nterror(req, nt_status);
                              return;
               }
+
+             if (W_ERROR_EQUAL(status, WERR_DS_NO_SUCH_OBJECT)) {
+                /* linked value to non existing object */
+                DEBUG(0,("try to skip chunk\n"));
+        }
                if (state->op->extended_op == DRSUAPI_EXOP_NONE) {
                              /* if it applied fine, we need to update the
highwatermark */
----------------------------------------------------------------------------------------------------------
Andrej Gessel (andrej.gessel at janztec.com<mailto:andrej.gessel at
janztec.com>)
Software Entwicklung

On Tue, 2017-06-13 at 12:01 +0000, Andrej Gessel via samba
wrote:
> Hello everyone,
> 
> i try to use Samba RODC(4.6.5) with W2K8R2. Windows AD has around
> 35000 objects. My Samba machine is small one (ARM 32bit CPU) with
> only 2GB physical memory, so i can’t join to the domain because of
> expensive memory usage.
> To solve this Problem, i decide to replicate only critical objects
> and then let samba_kcc to get other objects.
> 
> 1 ) Is this an possible way to use Samba AD or should i replicate all
> objects while joining new ad?
> 
> As result i got fast join and no more problems with memory usage.
> But i got some other errors for example:
> 
> ../source4/dsdb/repl/replicated_objects.c:933 Failed to prepare
> commit of transaction: DRS linked attribute for GUID 6709d915-7dfb-
> 41bf-b695-dd1a798d7718 - DN not found
> 
> Object with this objectGUID exists in AD, but was not replicated yet.
> So i create to patches follow to skip the chunks with „problem“-
> objects and try to get the other missing objects with effect, that
> links can be created and i can replicate rest.
> 
> 2) Is it rigth way to raise replication on such errors? Maybe it is
> better to skip the chunk and replicate other stuff first and let
> samba_kcc do his work and repeat the skiped chunks?
Tim Beale (CC'ed) is working to improve the Samba behaviour here,
hopefully for Samba 4.7.  It isn't safe to skip objects or links, as
you won't ever get them again.  This can lead to ongoing replication
corruption. 

A full sync would tend to fix it up again, but I can't suggest this
approach.

We know there are dragons here!

Andrew Bartlett
-- 
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT   
https://catalyst.net.nz/services/samba

Hi Andrej,

I've been testing DRS on Windows a bit and I noticed that Windows
replication can send the linked attribute before it sends the source
object, which is the problem you're seeing.

The MS-DRSR spec says that in this case the client should resend the
GetNCChanges request with the GET_ANC flag set. In my testing this
resolves the problem - Windows will include the source object for the
linked attribute in the same replication chunk.

This problem doesn't happen with Samba-to-Samba replication, because the
source object for the linked attribute is guaranteed to have already
been sent. However, Samba does already have support to retry
GetNCChanges with GET_ANC if it encounters a missing parent. Basically,
the solution would just be a matter of propagating the
WERR_DS_DRA_MISSING_PARENT error back to the
dsdb_replicated_objects_commit() caller in your 'DN not found' case as
well.

I'm currently working on a patch to do something similar when the linked
attribute's target object is missing (you will probably hit this case
too, sooner or later). If I have time, then I'll fix up the missing
source object case too.

Regards,
Tim Beale

On 14/06/17 11:34, Andrew Bartlett via samba wrote:
> On Tue, 2017-06-13 at 12:01 +0000, Andrej Gessel via samba wrote:
>> Hello everyone,
>>
>> i try to use Samba RODC(4.6.5) with W2K8R2. Windows AD has around
>> 35000 objects. My Samba machine is small one (ARM 32bit CPU) with
>> only 2GB physical memory, so i can’t join to the domain because of
>> expensive memory usage.
>> To solve this Problem, i decide to replicate only critical objects
>> and then let samba_kcc to get other objects.
>>
>> 1 ) Is this an possible way to use Samba AD or should i replicate all
>> objects while joining new ad?
>>
>> As result i got fast join and no more problems with memory usage.
>> But i got some other errors for example:
>>
>> ../source4/dsdb/repl/replicated_objects.c:933 Failed to prepare
>> commit of transaction: DRS linked attribute for GUID 6709d915-7dfb-
>> 41bf-b695-dd1a798d7718 - DN not found
>>
>> Object with this objectGUID exists in AD, but was not replicated yet.
>> So i create to patches follow to skip the chunks with „problem“-
>> objects and try to get the other missing objects with effect, that
>> links can be created and i can replicate rest.
>>
>> 2) Is it rigth way to raise replication on such errors? Maybe it is
>> better to skip the chunk and replicate other stuff first and let
>> samba_kcc do his work and repeat the skiped chunks?
> Tim Beale (CC'ed) is working to improve the Samba behaviour here,
> hopefully for Samba 4.7.  It isn't safe to skip objects or links, as
> you won't ever get them again.  This can lead to ongoing replication
> corruption. 
>
> A full sync would tend to fix it up again, but I can't suggest this
> approach.
>
> We know there are dragons here!
>
> Andrew Bartlett