samba - Feb 2020 - Ldapsearch against Samba AD returns records outside the search base

On Mon, 2020-02-03 at 18:17 +0200, Palle Kuling via samba
wrote:
> Hello,
> 
> I did some detective work here, stepping through all the versions
> from 
> the old 4.9.4 database onwards, building them from source on an
> isolated 
> system and doing ldapsearch against them. It is the change from
> 4.10.13 
> to 4.11.0 (or maybe in general from pre-4.11 to 4.11?) that breaks
> it; 
> after that the onelevel scope is not applied correctly.
Thanks.  That is where I would expect the issue to have come up.  We
did some pretty big changes to LDB and and LDAP server during that
period.

If you have the time, moving to git bisect as the tool and running
between samba-4.10.0rc1 and samba-4.11.0 would be awesome.
> Ldbsearch also returns wrong results when used with your commands 
Great, that rules out some odd client-specific (eg ASN.1 parsing)
issues and makes it a little easier for me to test.
> 
> samba-4.11.0$ sudo /usr/local/samba/bin/ldbsearch -H 
> ldb:///usr/local/samba/private/sam.ldb -s one -b 
> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
> -Uusername
> # record 1
> dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
> <snip>
> distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
> 
> # returned 1 records
> # 1 entries
> # 0 referrals
> 
> 
> Also, it seems that I was wrong about ldbsearch directly against the 
> backend DB working - it is simply because I forgot to use the
"one"
> scope, which seems to be the culprit here:
> 
> /usr/local/samba/private/sam.ldb.d# ldbsearch -H 
> DC\=INTERNAL\,DC\=XXX\,DC\=YY.ldb -b 
> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
> # returned 0 records
> # 0 entries
> # 0 referrals
> 
> /usr/local/samba/private/sam.ldb.d# ldbsearch -H 
> DC\=INTERNAL\,DC\=XXX\,DC\=YY.ldb -s one -b 
> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
> # record 1
> dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
> <snip>
> distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
Very interesting.  This does help narrow things down.
> # returned 1 records
> # 1 entries
> # 0 referrals
> 
> In order to test whether it happens on a joined DC or not, I need to 
> spin off some isolated test VM:s, so I'd have to come back on that in
> a 
> few days.
Thank you so much!

Andrew Bartlett

-- 
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT - Expert Open Source
Solutions
https://catalyst.net.nz/services/samba

Hello,

The problem is present also in a joined DC; tested with ldbsearch both 
against ldaps and the sam.ldb directly. Used a fresh build of version 
4.11.4 to test and joined it against a 4.11.4 clone. Samba-tool ldapcmp 
shows no differences and replication is working. Is there something else 
that would make sense to test, like joining 4.11 series against a 
pre-4.11 where the problem is not present or similar?

I don't know if this is of any value, but the directory is around five 
years old, originally created on maybe 4.1 or 4.2 series. It has seen 
some format changes during upgrades along the way. Maybe that could 
explain why Rowland has trouble re-creating the issue?

Regards,
-P

On 2020-02-04 00:08, Andrew Bartlett wrote:
> On Mon, 2020-02-03 at 18:17 +0200, Palle Kuling via samba wrote:
>> Hello,
>> 
>> I did some detective work here, stepping through all the versions
>> from
>> the old 4.9.4 database onwards, building them from source on an
>> isolated
>> system and doing ldapsearch against them. It is the change from
>> 4.10.13
>> to 4.11.0 (or maybe in general from pre-4.11 to 4.11?) that breaks
>> it;
>> after that the onelevel scope is not applied correctly.
> 
> Thanks.  That is where I would expect the issue to have come up.  We
> did some pretty big changes to LDB and and LDAP server during that
> period.
> 
> If you have the time, moving to git bisect as the tool and running
> between samba-4.10.0rc1 and samba-4.11.0 would be awesome.
> 
>> Ldbsearch also returns wrong results when used with your commands
> 
> Great, that rules out some odd client-specific (eg ASN.1 parsing)
> issues and makes it a little easier for me to test.
> 
>> 
>> samba-4.11.0$ sudo /usr/local/samba/bin/ldbsearch -H
>> ldb:///usr/local/samba/private/sam.ldb -s one -b
>> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
>> -Uusername
>> # record 1
>> dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
>> <snip>
>> distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
>> 
>> # returned 1 records
>> # 1 entries
>> # 0 referrals
>> 
>> 
>> Also, it seems that I was wrong about ldbsearch directly against the
>> backend DB working - it is simply because I forgot to use the
"one"
>> scope, which seems to be the culprit here:
>> 
>> /usr/local/samba/private/sam.ldb.d# ldbsearch -H
>> DC\=INTERNAL\,DC\=XXX\,DC\=YY.ldb -b
>> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
>> # returned 0 records
>> # 0 entries
>> # 0 referrals
>> 
>> /usr/local/samba/private/sam.ldb.d# ldbsearch -H
>> DC\=INTERNAL\,DC\=XXX\,DC\=YY.ldb -s one -b
>> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
>> # record 1
>> dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
>> <snip>
>> distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
> 
> Very interesting.  This does help narrow things down.
> 
>> # returned 1 records
>> # 1 entries
>> # 0 referrals
>> 
>> In order to test whether it happens on a joined DC or not, I need to
>> spin off some isolated test VM:s, so I'd have to come back on that
in
>> a
>> few days.
> 
> Thank you so much!
> 
> Andrew Bartlett

Hello,

I did a git bisect between 4.10.0rc1 and 4.11.0. The result is as 
follows:
b6b5b5fe355fee2a4096e9214831cb88c7a2a4c6 is the first bad commit
Date:   Wed Mar 6 15:28:45 2019 +1300

     lib ldb key value: fix index buffering

Is there anything else I should check?

Regards,
-P

On 2020-02-04 00:08, Andrew Bartlett via samba wrote:
> On Mon, 2020-02-03 at 18:17 +0200, Palle Kuling via samba wrote:
>> Hello,
>> 
>> I did some detective work here, stepping through all the versions
>> from
>> the old 4.9.4 database onwards, building them from source on an
>> isolated
>> system and doing ldapsearch against them. It is the change from
>> 4.10.13
>> to 4.11.0 (or maybe in general from pre-4.11 to 4.11?) that breaks
>> it;
>> after that the onelevel scope is not applied correctly.
> 
> Thanks.  That is where I would expect the issue to have come up.  We
> did some pretty big changes to LDB and and LDAP server during that
> period.
> 
> If you have the time, moving to git bisect as the tool and running
> between samba-4.10.0rc1 and samba-4.11.0 would be awesome.
> 
>> Ldbsearch also returns wrong results when used with your commands
> 
> Great, that rules out some odd client-specific (eg ASN.1 parsing)
> issues and makes it a little easier for me to test.
> 
>> 
>> samba-4.11.0$ sudo /usr/local/samba/bin/ldbsearch -H
>> ldb:///usr/local/samba/private/sam.ldb -s one -b
>> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
>> -Uusername
>> # record 1
>> dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
>> <snip>
>> distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
>> 
>> # returned 1 records
>> # 1 entries
>> # 0 referrals
>> 
>> 
>> Also, it seems that I was wrong about ldbsearch directly against the
>> backend DB working - it is simply because I forgot to use the
"one"
>> scope, which seems to be the culprit here:
>> 
>> /usr/local/samba/private/sam.ldb.d# ldbsearch -H
>> DC\=INTERNAL\,DC\=XXX\,DC\=YY.ldb -b
>> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
>> # returned 0 records
>> # 0 entries
>> # 0 referrals
>> 
>> /usr/local/samba/private/sam.ldb.d# ldbsearch -H
>> DC\=INTERNAL\,DC\=XXX\,DC\=YY.ldb -s one -b
>> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
>> # record 1
>> dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
>> <snip>
>> distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
> 
> Very interesting.  This does help narrow things down.
> 
>> # returned 1 records
>> # 1 entries
>> # 0 referrals
>> 
>> In order to test whether it happens on a joined DC or not, I need to
>> spin off some isolated test VM:s, so I'd have to come back on that
in
>> a
>> few days.
> 
> Thank you so much!
> 
> Andrew Bartlett
> 
> --
> Andrew Bartlett
> https://samba.org/~abartlet/
> Authentication Developer, Samba Team         https://samba.org
> Samba Development and Support, Catalyst IT - Expert Open Source
> Solutions
> https://catalyst.net.nz/services/samba

On Thu, 2020-02-06 at 16:21 +0200, Palle Kuling via samba
wrote:
> Hello,
> 
> I did a git bisect between 4.10.0rc1 and 4.11.0. The result is as 
> follows:
> b6b5b5fe355fee2a4096e9214831cb88c7a2a4c6 is the first bad commit
> Date:   Wed Mar 6 15:28:45 2019 +1300
> 
>      lib ldb key value: fix index buffering
> 
> Is there anything else I should check?
Thanks, that helps a lot.  I'll be looking at this today and next week.

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba