samba - Mar 2018 - Unable to successfully join Samba 4.8.0 or Windows 2008 R2 to a Samba 4.6.7 DC, unable to upgrade in-place

On Thu, 2018-03-15 at 16:10 -0400, Justin Foreman wrote:
> Okay. The master build worked on the join. Excellent. So it was 4.8.0
having a problem.
> 
> I’ve attempted to join 2008 R2 now to 4.8.0 but it still blue screens.
Should I start a new thread for clarity’s sake?
So, I think this, the upgrade issues and even the old
servicePrincipalName handling bug is all the same thing.

What happened is that in the past we did not reject:

servicePrincipalName: HOST/foo
servicePrincipalName: host/foo

Then with 4.8 the index code rejected this.  With master for 4.9 we
decided it wasn't the index code's job to do this, so fixed that (it
helped another use case). 

Add to this the upgrade code for GUID indexes didn't assert that if the
re-index failed that we must abort the transaction, so the partial
upgrade case gets committed and it all dies on the next DB open.

At least that is the theory I'll be working to prove or disprove on
Monday.

I think the core short-term fix is in:

commit 5c1504b94d1417894176811f18c5d450de22cfd2
Author: Gary Lockyer <gary at catalyst.net.nz>
Date:   Wed Feb 28 11:47:22 2018 +1300

    ldb_tdb: Do not fail in GUID index mode if there is a duplicate
attribute
    
    It is not the job of the index code to enforce this, but do give a
    a warning given it has been detected.
    
    However, now that we do allow it, we must never return the same
    object twice to the caller, so filter for it in
ltdb_index_filter().
    
    The GUID list is sorted, which makes this cheap to handle,
thankfully.
    
    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
    Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
    Reviewed-by: Andrew Bartlett <abartlet at samba.org>


The replication to windows may be failing because of the duplicate
value, eventually we will need to write a dbcheck rule to fix that.

Thanks all for your patience!

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

On Sat, 2018-03-17 at 07:34 +1300, Andrew Bartlett via samba-technical
wrote:
> On Thu, 2018-03-15 at 16:10 -0400, Justin Foreman wrote:
> > Okay. The master build worked on the join. Excellent. So it was 4.8.0
having a problem.
> > 
> > I’ve attempted to join 2008 R2 now to 4.8.0 but it still blue screens.
Should I start a new thread for clarity’s sake?
> 
> So, I think this, the upgrade issues and even the old
> servicePrincipalName handling bug is all the same thing.
> 
> What happened is that in the past we did not reject:
> 
> servicePrincipalName: HOST/foo
> servicePrincipalName: host/foo
To further tie threads together, that would be:
https://bugzilla.samba.org/show_bug.cgi?id=8929
> Then with 4.8 the index code rejected this.  With master for 4.9 we
> decided it wasn't the index code's job to do this, so fixed that
(it
> helped another use case). 
> 
> Add to this the upgrade code for GUID indexes didn't assert that if the
> re-index failed that we must abort the transaction, so the partial
> upgrade case gets committed and it all dies on the next DB open.
> 
> At least that is the theory I'll be working to prove or disprove on
> Monday.
> 
> I think the core short-term fix is in:
> 
> commit 5c1504b94d1417894176811f18c5d450de22cfd2
> Author: Gary Lockyer <gary at catalyst.net.nz>
> Date:   Wed Feb 28 11:47:22 2018 +1300
> 
>     ldb_tdb: Do not fail in GUID index mode if there is a duplicate
> attribute
>     
>     It is not the job of the index code to enforce this, but do give a
>     a warning given it has been detected.
>     
>     However, now that we do allow it, we must never return the same
>     object twice to the caller, so filter for it in
> ltdb_index_filter().
>     
>     The GUID list is sorted, which makes this cheap to handle,
> thankfully.
>     
>     Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
>     Reviewed-by: Douglas Bagnall <douglas.bagnall at catalyst.net.nz>
>     Reviewed-by: Andrew Bartlett <abartlet at samba.org>
> 
> 
> The replication to windows may be failing because of the duplicate
> value, eventually we will need to write a dbcheck rule to fix that.
> 
> Thanks all for your patience!
> 
> Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

Is there a sensible way for me to manually remove the duplicate SPNs in the mean
time?

Justin
> On Mar 17, 2018, at 3:09 PM, Andrew Bartlett <abartlet at samba.org>
wrote:
> 
> On Sat, 2018-03-17 at 07:34 +1300, Andrew Bartlett via samba-technical
> wrote:
>> On Thu, 2018-03-15 at 16:10 -0400, Justin Foreman wrote:
>>> Okay. The master build worked on the join. Excellent. So it was
4.8.0 having a problem.
>>> 
>>> I’ve attempted to join 2008 R2 now to 4.8.0 but it still blue
screens. Should I start a new thread for clarity’s sake?
>> 
>> So, I think this, the upgrade issues and even the old
>> servicePrincipalName handling bug is all the same thing.
>> 
>> What happened is that in the past we did not reject:
>> 
>> servicePrincipalName: HOST/foo
>> servicePrincipalName: host/foo
> 
> To further tie threads together, that would be:
> https://bugzilla.samba.org/show_bug.cgi?id=8929
> 
>> Then with 4.8 the index code rejected this.  With master for 4.9 we
>> decided it wasn't the index code's job to do this, so fixed
that (it
>> helped another use case). 
>> 
>> Add to this the upgrade code for GUID indexes didn't assert that if
the
>> re-index failed that we must abort the transaction, so the partial
>> upgrade case gets committed and it all dies on the next DB open.
>> 
>> At least that is the theory I'll be working to prove or disprove on
>> Monday.
>> 
>> I think the core short-term fix is in:
>> 
>> commit 5c1504b94d1417894176811f18c5d450de22cfd2
>> Author: Gary Lockyer <gary at catalyst.net.nz>
>> Date:   Wed Feb 28 11:47:22 2018 +1300
>> 
>>    ldb_tdb: Do not fail in GUID index mode if there is a duplicate
>> attribute
>> 
>>    It is not the job of the index code to enforce this, but do give a
>>    a warning given it has been detected.
>> 
>>    However, now that we do allow it, we must never return the same
>>    object twice to the caller, so filter for it in
>> ltdb_index_filter().
>> 
>>    The GUID list is sorted, which makes this cheap to handle,
>> thankfully.
>> 
>>    Signed-off-by: Gary Lockyer <gary at catalyst.net.nz>
>>    Reviewed-by: Douglas Bagnall <douglas.bagnall at
catalyst.net.nz>
>>    Reviewed-by: Andrew Bartlett <abartlet at samba.org>
>> 
>> 
>> The replication to windows may be failing because of the duplicate
>> value, eventually we will need to write a dbcheck rule to fix that.
>> 
>> Thanks all for your patience!
>> 
>> Andrew Bartlett
> -- 
> Andrew Bartlett                       http://samba.org/~abartlet/
> Authentication Developer, Samba Team  http://samba.org
> Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba
>