samba - Feb 2020 - Ldapsearch against Samba AD returns records outside the search base

On Fri, 2020-01-31 at 15:50 +0200, Palle Kuling via samba
wrote:
> Hi,
> 
> I noticed the following problem with records returned outside the search 
> base when the query is run against a Samba DC, but when the same query 
> is run against a Windows 2008 or 2012 DC it does not happen. I'm pretty
> sure it worked correctly in the past. I updated from Samba 4.9.4 to 
> 4.11.4 in December, but I noticed it only today, and I no longer have a 
> backup of the old installation to verify. I tried building versions 
> 4.11.5 and 4.11.6 against the same database, but they all behave in the 
> same way. Am I missing some config option, or is it a bug? 
Something as fundamental as this can't be configured, this is a
worrying bug.
> These kinds 
> of queries are used to check if an account exists in a certain OU, so I 
> would not want the DC:s to behave differently for the same query.
Indeed.
> This is how it looks when I run a query (I redacted the domain and 
> account names a bit):
> 
> ldapsearch -D username at internal.xxx.yy -w password -H ldaps://<samba
DC>
> -s one -b ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
> # extended LDIF
> #
> # LDAPv3
> # base <ou=business,dc=internal,dc=xxx,dc=yy> with scope oneLevel
> # filter: samaccountname=testadmin
> # requesting: ALL
> #
This is a worry.  Can you file a bug?  I've sent you an invite to our
bugzilla.  It seems we have an issue here applying the 'onelevel'
restriction.

There have been some pretty major changes (which allowed Samba's AD DC
to grow to the scale and performance it now has) between 4.9 and 4.11,
and I'm very keen to learn as much as possible about this bug.

To be clear, I consider this a very serious regression, you absoultly
should expect Samba to honour the LDAP specification and match Windows
AD behaviour!

That you can reproduce it with ldapsearch is awesome.  Does it
reproduct with ldbsearch?  Please run it like this:

ldbsearch -H ldaps://<samba DC> -s one -b
ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin -U$USER

ldbsearch -H ldb://usr/local/samba/private/sam.ldb -s one -b
ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin -U$USER

I ask this not to discount your bug, just to isolate layers.  I notice
in your thread with Rowland then you did a search against the backend
DB which behaved correctly.  That is useful information, as the 'AD'
semantics are invoked one layer up from there, on sam.ldb.

Finally, does it happen if you join a new DC to the domain, or only on
your upgraded DC?

If it does happen on a joined DC, and you can create a new lab domain
and reproduce it there it would be awesome, and give us a safe place to
test more theories:

https://wiki.samba.org/index.php/Create_a_samba_lab-domain

In particular, if you could join older versions of Samba to that lab
domain, and perform a bisect to find where this was broken, this would
be awesome.   If you could do a git bisect down to the commit that
would be even better, if that is within your capability.

If for some reason rejoining is a problem (it can be tedious in a
bisect), just be aware of these notes on downgrading the DB in-place:

https://wiki.samba.org/index.php/Downgrading_an_Active_Directory_DC

Thank you so much for reporting this issue, I hope we get to the bottom
soon!

Andrew Bartlett
-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

Hello,

I did some detective work here, stepping through all the versions from 
the old 4.9.4 database onwards, building them from source on an isolated 
system and doing ldapsearch against them. It is the change from 4.10.13 
to 4.11.0 (or maybe in general from pre-4.11 to 4.11?) that breaks it; 
after that the onelevel scope is not applied correctly.

Ldbsearch also returns wrong results when used with your commands (it 
took me a while to figure out that I needed "tls verify peer =
no_check"
and "ldap server require strong auth = no" to be able to run the
query):

samba-4.11.0$ /usr/local/samba/bin/ldbsearch -H ldaps://dc01 -s one -b 
ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin -Uusername
Password for [XXX\username]:
# record 1
dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
<snip>
distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy

# returned 1 records
# 1 entries
# 0 referrals


samba-4.11.0$ sudo /usr/local/samba/bin/ldbsearch -H 
ldb:///usr/local/samba/private/sam.ldb -s one -b 
ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin -Uusername
# record 1
dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
<snip>
distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy

# returned 1 records
# 1 entries
# 0 referrals


Also, it seems that I was wrong about ldbsearch directly against the 
backend DB working - it is simply because I forgot to use the "one" 
scope, which seems to be the culprit here:

/usr/local/samba/private/sam.ldb.d# ldbsearch -H 
DC\=INTERNAL\,DC\=XXX\,DC\=YY.ldb -b 
ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
# returned 0 records
# 0 entries
# 0 referrals

/usr/local/samba/private/sam.ldb.d# ldbsearch -H 
DC\=INTERNAL\,DC\=XXX\,DC\=YY.ldb -s one -b 
ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
# record 1
dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
<snip>
distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy

# returned 1 records
# 1 entries
# 0 referrals

In order to test whether it happens on a joined DC or not, I need to 
spin off some isolated test VM:s, so I'd have to come back on that in a 
few days.

Regards,
-P

On 2020-02-01 21:20, Andrew Bartlett wrote:
> On Fri, 2020-01-31 at 15:50 +0200, Palle Kuling via samba wrote:
>> Hi,
>> 
>> I noticed the following problem with records returned outside the 
>> search
>> base when the query is run against a Samba DC, but when the same query
>> is run against a Windows 2008 or 2012 DC it does not happen. I'm 
>> pretty
>> sure it worked correctly in the past. I updated from Samba 4.9.4 to
>> 4.11.4 in December, but I noticed it only today, and I no longer have 
>> a
>> backup of the old installation to verify. I tried building versions
>> 4.11.5 and 4.11.6 against the same database, but they all behave in 
>> the
>> same way. Am I missing some config option, or is it a bug?
> 
> Something as fundamental as this can't be configured, this is a
> worrying bug.
> 
>> These kinds
>> of queries are used to check if an account exists in a certain OU, so 
>> I
>> would not want the DC:s to behave differently for the same query.
> 
> Indeed.
> 
>> This is how it looks when I run a query (I redacted the domain and
>> account names a bit):
>> 
>> ldapsearch -D username at internal.xxx.yy -w password -H
ldaps://<samba
>> DC>
>> -s one -b ou=business,dc=internal,dc=xxx,dc=yy 
>> samaccountname=testadmin
>> # extended LDIF
>> #
>> # LDAPv3
>> # base <ou=business,dc=internal,dc=xxx,dc=yy> with scope oneLevel
>> # filter: samaccountname=testadmin
>> # requesting: ALL
>> #
> 
> This is a worry.  Can you file a bug?  I've sent you an invite to our
> bugzilla.  It seems we have an issue here applying the 'onelevel'
> restriction.
> 
> There have been some pretty major changes (which allowed Samba's AD DC
> to grow to the scale and performance it now has) between 4.9 and 4.11,
> and I'm very keen to learn as much as possible about this bug.
> 
> To be clear, I consider this a very serious regression, you absoultly
> should expect Samba to honour the LDAP specification and match Windows
> AD behaviour!
> 
> That you can reproduce it with ldapsearch is awesome.  Does it
> reproduct with ldbsearch?  Please run it like this:
> 
> ldbsearch -H ldaps://<samba DC> -s one -b
> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin -U$USER
> 
> ldbsearch -H ldb://usr/local/samba/private/sam.ldb -s one -b
> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin -U$USER
> 
> I ask this not to discount your bug, just to isolate layers.  I notice
> in your thread with Rowland then you did a search against the backend
> DB which behaved correctly.  That is useful information, as the
'AD'
> semantics are invoked one layer up from there, on sam.ldb.
> 
> Finally, does it happen if you join a new DC to the domain, or only on
> your upgraded DC?
> 
> If it does happen on a joined DC, and you can create a new lab domain
> and reproduce it there it would be awesome, and give us a safe place to
> test more theories:
> 
> https://wiki.samba.org/index.php/Create_a_samba_lab-domain
> 
> In particular, if you could join older versions of Samba to that lab
> domain, and perform a bisect to find where this was broken, this would
> be awesome.   If you could do a git bisect down to the commit that
> would be even better, if that is within your capability.
> 
> If for some reason rejoining is a problem (it can be tedious in a
> bisect), just be aware of these notes on downgrading the DB in-place:
> 
> https://wiki.samba.org/index.php/Downgrading_an_Active_Directory_DC
> 
> Thank you so much for reporting this issue, I hope we get to the bottom
> soon!
> 
> Andrew Bartlett

On 03/02/2020 16:17, Palle Kuling via samba wrote:
> Hello,
>
> I did some detective work here, stepping through all the versions from 
> the old 4.9.4 database onwards, building them from source on an 
> isolated system and doing ldapsearch against them. It is the change 
> from 4.10.13 to 4.11.0 (or maybe in general from pre-4.11 to 4.11?) 
> that breaks it; after that the onelevel scope is not applied correctly.
>
> Ldbsearch also returns wrong results when used with your commands (it 
> took me a while to figure out that I needed "tls verify peer = 
> no_check" and "ldap server require strong auth = no" to be
able to run
> the query):
>
> samba-4.11.0$ /usr/local/samba/bin/ldbsearch -H ldaps://dc01 -s one -b 
> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin -Uusername
> Password for [XXX\username]:
> # record 1
> dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
> <snip>
> distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
>
> # returned 1 records
> # 1 entries
> # 0 referrals
>
>
> samba-4.11.0$ sudo /usr/local/samba/bin/ldbsearch -H 
> ldb:///usr/local/samba/private/sam.ldb -s one -b 
> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin -Uusername
> # record 1
> dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
> <snip>
> distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
>
> # returned 1 records
> # 1 entries
> # 0 referrals
>
>
> Also, it seems that I was wrong about ldbsearch directly against the 
> backend DB working - it is simply because I forgot to use the
"one"
> scope, which seems to be the culprit here:
>
> /usr/local/samba/private/sam.ldb.d# ldbsearch -H 
> DC\=INTERNAL\,DC\=XXX\,DC\=YY.ldb -b 
> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
> # returned 0 records
> # 0 entries
> # 0 referrals
>
> /usr/local/samba/private/sam.ldb.d# ldbsearch -H 
> DC\=INTERNAL\,DC\=XXX\,DC\=YY.ldb -s one -b 
> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
> # record 1
> dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
> <snip>
> distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
>
> # returned 1 records
> # 1 entries
> # 0 referrals
>
> In order to test whether it happens on a joined DC or not, I need to 
> spin off some isolated test VM:s, so I'd have to come back on that in 
> a few days.
>
> Regards,
> -P
This is where I differ from you, using your search command from your 
original post (altered for my domain), I always get the expected result. 
I have tested this on a few Samba versions, all of them from Louis's repo.

Rowland

On Mon, 2020-02-03 at 18:17 +0200, Palle Kuling via samba
wrote:
> Hello,
> 
> I did some detective work here, stepping through all the versions
> from 
> the old 4.9.4 database onwards, building them from source on an
> isolated 
> system and doing ldapsearch against them. It is the change from
> 4.10.13 
> to 4.11.0 (or maybe in general from pre-4.11 to 4.11?) that breaks
> it; 
> after that the onelevel scope is not applied correctly.
Thanks.  That is where I would expect the issue to have come up.  We
did some pretty big changes to LDB and and LDAP server during that
period.

If you have the time, moving to git bisect as the tool and running
between samba-4.10.0rc1 and samba-4.11.0 would be awesome.
> Ldbsearch also returns wrong results when used with your commands 
Great, that rules out some odd client-specific (eg ASN.1 parsing)
issues and makes it a little easier for me to test.
> 
> samba-4.11.0$ sudo /usr/local/samba/bin/ldbsearch -H 
> ldb:///usr/local/samba/private/sam.ldb -s one -b 
> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
> -Uusername
> # record 1
> dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
> <snip>
> distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
> 
> # returned 1 records
> # 1 entries
> # 0 referrals
> 
> 
> Also, it seems that I was wrong about ldbsearch directly against the 
> backend DB working - it is simply because I forgot to use the
"one"
> scope, which seems to be the culprit here:
> 
> /usr/local/samba/private/sam.ldb.d# ldbsearch -H 
> DC\=INTERNAL\,DC\=XXX\,DC\=YY.ldb -b 
> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
> # returned 0 records
> # 0 entries
> # 0 referrals
> 
> /usr/local/samba/private/sam.ldb.d# ldbsearch -H 
> DC\=INTERNAL\,DC\=XXX\,DC\=YY.ldb -s one -b 
> ou=business,dc=internal,dc=xxx,dc=yy samaccountname=testadmin
> # record 1
> dn: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
> <snip>
> distinguishedName: CN=Test Admin,OU=Test,DC=internal,DC=xxx,DC=yy
Very interesting.  This does help narrow things down.
> # returned 1 records
> # 1 entries
> # 0 referrals
> 
> In order to test whether it happens on a joined DC or not, I need to 
> spin off some isolated test VM:s, so I'd have to come back on that in
> a 
> few days.
Thank you so much!

Andrew Bartlett

-- 
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team         https://samba.org
Samba Development and Support, Catalyst IT - Expert Open Source
Solutions
https://catalyst.net.nz/services/samba