thr3ads.net - samba - [Samba] High cpu load on LDAP [Nov 2024]

If this information is useful, please help other people find it:
Share via:

Heinz Hölzl

2024-Nov-18 13:29 UTC

[Samba] High cpu load on LDAP

hi,

I have to activate the thread again ...

we keep having preformance problems on the DC, especially on Monday
morning when the PCs are switched on and the users log in.

some ldap-searches take a very long time, sometimes even over 15
seconds

e.g:

ldapsrv_SearchRequest: LDAP Query: Duration was 15.74s, SearchRequest
by S-1-5-21-xxxxxxxxxx-xxxxxxxxxxxxxx-8585 from
ipv4:192.168.35.117:49240 filter: [(objectClass=user)] basedn:
[DC=example,DC=net] scope: [SUB] result: Success

The load of the ldap processes reaches 100% of a CPU. The ldapserver is
then no longer responsive.

It seems that the ldapsearches are blocking each other.

The result is very long response times for login and other
authentications.

we have 6 DC, approx. 10,000 active users and approx. 5000 PC, 10 samba
fileservers, all with current samba version

is there a way to increase the performance here?

regards,
Heinz

Here are more logs (at a very quiet time) : 

  ldapsrv_SearchRequest: LDAP Query: Duration was 1.74s, SearchRequest
by S-1-5-21-xxxxxxxxxxxxx-xxxxxxxxxxxxxx-xxxxxxxxxxxxxxx-3084 from
ipv4:192.168.48.87:33768 filter:
[(|(objectClass=user)(objectClass=group))] basedn: [dc=GVCC,dc=NET]
scope: [SUB] result: Success
  ldapsrv_SearchRequest: LDAP Query: Duration was 1.79s, SearchRequest
by S-1-5-21-xxxxxxxxxxxxx-xxxxxxxxxxxxxx-xxxxxxxxxxxxxxx-3084 from
ipv4:192.168.48.87:33768 filter:
[(|(objectClass=user)(objectClass=group))] basedn: [dc=GVCC,dc=NET]
scope: [SUB] result: Success
  ldapsrv_SearchRequest: LDAP Query: Duration was 0.63s, SearchRequest
by S-1-5-18 from unix: filter:
[(&(objectClass=user)(userAccountControl:1.2.840.113556.1.4.803:=512)(!
(sAMAccountName=krbtgt*)))] basedn: [DC=example,DC=net] scope: [SUB]
result: Success
  ldapsrv_SearchRequest: LDAP Query: Duration was 1.80s, SearchRequest
by S-1-5-21-xxxxxxxxxxxxx-xxxxxxxxxxxxxx-xxxxxxxxxxxxxxx-3048 from
ipv4:192.168.19.22:54708 filter:
[(|(objectClass=user)(objectClass=group))] basedn: [dc=GVCC,dc=NET]
scope: [SUB] result: Success
  ldapsrv_SearchRequest: LDAP Query: Duration was 1.74s, SearchRequest
by S-1-5-21-xxxxxxxxxxxxx-xxxxxxxxxxxxxx-xxxxxxxxxxxxxxx-3048 from
ipv4:192.168.19.22:54708 filter:
[(|(objectClass=user)(objectClass=group))] basedn: [dc=GVCC,dc=NET]
scope: [SUB] result: Success
  ldapsrv_SearchRequest: LDAP Query: Duration was 0.67s, SearchRequest
by S-1-5-18 from unix: filter:
[(&(objectClass=user)(userAccountControl:1.2.840.113556.1.4.803:=512)(!
(sAMAccountName=krbtgt*)))] basedn: [DC=example,DC=net] scope: [SUB]
result: Success
  ldapsrv_SearchRequest: LDAP Query: Duration was 0.64s, SearchRequest
by S-1-5-18 from unix: filter:
[(&(objectClass=user)(userAccountControl:1.2.840.113556.1.4.803:=512)(!
(sAMAccountName=krbtgt*)))] basedn: [DC=example,DC=net] scope: [SUB]
result: Success
  ldapsrv_SearchRequest: LDAP Query: Duration was 1.93s, SearchRequest
by S-1-5-21-xxxxxxxxxxxxx-xxxxxxxxxxxxxx-xxxxxxxxxxxxxxx-3085 from
ipv4:192.168.44.65:59148 filter:
[(|(objectClass=user)(objectClass=group))] basedn: [dc=GVCC,dc=NET]
scope: [SUB] result: Success
  ldapsrv_SearchRequest: LDAP Query: Duration was 1.88s, SearchRequest
by S-1-5-21-xxxxxxxxxxxxx-xxxxxxxxxxxxxx-xxxxxxxxxxxxxxx-3085 from
ipv4:192.168.44.65:59148 filter:
[(|(objectClass=user)(objectClass=group))] basedn: [dc=GVCC,dc=NET]
scope: [SUB] result: Success
  ldapsrv_SearchRequest: LDAP Query: Duration was 0.61s, SearchRequest
by S-1-5-18 from unix: filter:
[(&(objectClass=user)(userAccountControl:1.2.840.113556.1.4.803:=512)(!
(sAMAccountName=krbtgt*)))] basedn: [DC=example,DC=net] scope: [SUB]
result: Success
  ldapsrv_SearchRequest: LDAP Query: Duration was 1.83s, SearchRequest
by S-1-5-21-xxxxxxxxxxxxx-xxxxxxxxxxxxxx-xxxxxxxxxxxxxxx-3085 from
ipv4:192.168.44.65:59148 filter:
[(|(objectClass=user)(objectClass=group))] basedn: [dc=GVCC,dc=NET]
scope: [SUB] result: Success
  ldapsrv_SearchRequest: LDAP Query: Duration was 0.62s, SearchRequest
by S-1-5-18 from unix: filter:
[(&(objectClass=user)(userAccountControl:1.2.840.113556.1.4.803:=512)(!
(sAMAccountName=krbtgt*)))] basedn: [DC=example,DC=net] scope: [SUB]
result: Success
  ldapsrv_SearchRequest: LDAP Query: Duration was 0.69s, SearchRequest
by S-1-5-18 from unix: filter:
[(&(objectClass=user)(userAccountControl:1.2.840.113556.1.4.803:=512)(!
(sAMAccountName=krbtgt*)))] basedn: [DC=example,DC=net] scope: [SUB]
result: Success

Am Freitag, dem 30.09.2022 um 09:14 +1300 schrieb Andrew
Bartlett:> [Sie erhalten nicht h?ufig E-Mails von abartlet at samba.org. Weitere
> Informationen, warum dies wichtig ist, finden Sie unter
> https://aka.ms/LearnAboutSenderIdentification?]
> 
> On Mon, 2022-09-19 at 14:04 +0000, Heinz H?lzl via samba wrote:
> > hello,
> > I often have the problem of high load on the LDAP processes.
> > 1-3 LDAP processes cause 100% cpu load for approx. 10 sec. This
> > happens
> > regularly in intervals of 2-3 minutes.
> > How can I find out which client is causing this load and why?
> > How can I configure the logging to see who/what is causing the LDAP
> > process?
> > We have about 5000 users, 4000 clients in our AD with 4 DCs with
> > Samba
> > 4.16.4.
> > 
> > thanks
> 
> 
> I would also note the similarity with this other post on the mailing
> list.
> 
> If this is the same issue, the sockets to the client would be stuck
> in
> CLOSE_WAIT.
> 
> https://lists.samba.org/archive/samba/2022-September/241873.html
> 
> Can you, on the instance with the high-cpu use, at the time it is in
> that state, run (matching the commands our mailing list user has
> run):
> 
> Work out which PIDs are high CPU use:
> 
> netstat -avp | grep <PID>
> 
> gdb -p <PID>
> bt full
> 
> strace -p <PID>
> 
> as well as a FlameGraph:
> https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#Instructions
> 
> I would love to get to the bottom of this.
> Andrew Bartlett
> 
> 
> --
> Andrew Bartlett (he/him)?????? https://samba.org/~abartlet/
> Samba Team Member (since 2001) https://samba.org
> Samba Team Lead, Catalyst IT?? https://catalyst.net.nz/services/samba
> 
> Samba Development and Support, Catalyst IT - Expert Open Source
> Solutions
>

Douglas Bagnall

2024-Nov-19 22:04 UTC

head link

[Samba] High cpu load on LDAP

On 19/11/24 02:29, Heinz H?lzl via samba wrote:> hi,
> 
> I have to activate the thread again ...
> 
> we keep having preformance problems on the DC, especially on Monday
> morning when the PCs are switched on and the users log in.
> 
> some ldap-searches take a very long time, sometimes even over 15
> seconds
> 
> e.g:
> 
> ldapsrv_SearchRequest: LDAP Query: Duration was 15.74s, SearchRequest
> by S-1-5-21-xxxxxxxxxx-xxxxxxxxxxxxxx-8585 from
> ipv4:192.168.35.117:49240 filter: [(objectClass=user)] basedn:
> [DC=example,DC=net] scope: [SUB] result: Success
> 
> 
> The load of the ldap processes reaches 100% of a CPU. The ldapserver is
> then no longer responsive.
> 
> It seems that the ldapsearches are blocking each other.
> 
> The result is very long response times for login and other
> authentications.
> 
> we have 6 DC, approx. 10,000 active users and approx. 5000 PC, 10 samba
> fileservers, all with current samba version
> 
> 
> is there a way to increase the performance here?
> 
It would not hurt to delete about 6000 old machine accounts.

But it probably comes down to member searches on big groups, as 
discussed in this other thread:

https://lists.samba.org/archive/samba/2024-October/249848.html

The underlying problem is that linked attributes (like member) are not 
indexed.

Why are linked attributes not indexed? because that is what the 
Microsoft AD schema says.

Why does the AD schema say that? Perhaps because backlinks (like 
memberOf) are effectively an index, so an additional index seemed 
redundant. Perhaps Microsoft AD actually uses the backlinks as an index 
in linked attribute searches. Or perhaps Samba AD is used in different 
ways (e.g. with OpenLDAP slurping up all the memberships), so Microsoft 
does not often hit this problem.

It is possible to tell Samba to index 'member'. That would probably make
this problem vanish, and there's a reasonable chance there would be no 
ill effects. I don't know if anyone has actually tried that.

I would like to solve this as a development problem, but that depends on 
time and funding and won't help you now.

BTW, that last thread ended with:
> Am Freitag, dem 30.09.2022 um 09:14 +1300 schrieb Andrew Bartlett:
>> [Sie erhalten nicht h?ufig E-Mails von abartlet at samba.org. Weitere
>> Informationen, warum dies wichtig ist, finden Sie unter
>> https://aka.ms/LearnAboutSenderIdentification?]
>>
>> Work out which PIDs are high CPU use:
>>
>> netstat -avp | grep <PID>
>>
>> gdb -p <PID>
>> bt full
>>
>> strace -p <PID>
>>
>> as well as a FlameGraph:
>>
https://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#Instructions
>>
>> I would love to get to the bottom of this.
>> Andrew Bartlett
which would still be useful. Just a sampling of
>> gdb -p <PID>
>> bt full
would confirm where the server was spending its time. If `bt full` looks 
too revealing, you could use `bt` instead.

Douglas

Possibly Parallel Threads

Search for more apparently analagous threads

samba - Nov 2024 - High cpu load on LDAP

[Samba] High cpu load on LDAP

[Samba] High cpu load on LDAP

Possibly Parallel Threads