CentOS - Jan 2009 - Issues with Ldap client on Centos 5

Hi Friends,

We are running some of the Centos 5 32 bit, 5.2 64-bit systems. These
systems are ldap clients and the ldap server is Windows 2003 Server.
Sometimes 1 or 2 services on these servers sucks 100% cpu and the load
becomes high on the server.

Below is an example where one the httpd process was eating 100% cpu
and we took dump of this process

gcore  17711

Core was generated by `/usr/sbin/httpd'.

#0  0x00002ad1849cd997 in ldap_chase_v3referrals () from
/usr/lib64/libldap-2.3.so.0

(gdb) bt full

#0  0x00002ad1849cd997 in ldap_chase_v3referrals () from
/usr/lib64/libldap-2.3.so.0

No symbol table info available.

#1  0x00002ad1849bc4dd in ldap_msgdelete () from /usr/lib64/libldap-2.3.so.0

No symbol table info available.

#2  0x00002ad1849bceb0 in ldap_result () from /usr/lib64/libldap-2.3.so.0

No symbol table info available.


/etc/ldap.conf file



host dc.example.com
base ou=users,dc=example,dc=com
binddn cn=ldap,ou=extra accounts,dc=example,dc=com
bindpw QrQcepFKHR6wGNXu4
scope sub
ssl no
nss_base_passwd dc=example,dc=com?sub
nss_base_shadow dc=example,dc=com?sub
nss_base_group dc=example,dc=com?sub
nss_map_objectclass posixAccount user
nss_map_objectclass shadowAccount user
nss_map_attribute uid sAMAccountName
nss_map_attribute uidNumber UidNumber
nss_map_attribute gidNumber GidNumber
nss_map_attribute loginShell LoginShell
nss_map_attribute gecos name
nss_map_attribute userPassword unixUserPassword
nss_map_attribute homeDirectory unixHomeDirectory
nss_map_objectclass posixGroup Group
nss_map_attribute uniqueMember msSFU30PosixMember
nss_map_attribute cn cn
pam_login_attribute sAMAccountName
pam_filter objectclass=user
pam_password md5
timelimit 0
sizelimit 0
tls_cacertdir /etc/openldap/cacerts

There are 2 bugs listed on the redhat site but no solution for this
problem has been provided.


https://bugzilla.redhat.com/show_bug.cgi?id=222667

https://bugzilla.redhat.com/show_bug.cgi?id=474181



Thanks & Regards

Ankush

Hello,

On Thu, Jan 22, 2009 at 01:41, ankush grover <ankushcentos at gmail.com>
wrote:
> We are running some of the Centos 5 32 bit, 5.2 64-bit systems. These
> systems are ldap clients and the ldap server is Windows 2003 Server.
How exactly? Are you using nss_ldap to get user ids from AD? Are you
authenticating to AD using LDAP? What are the lines that contain
"ldap" in your /etc/nsswitch.conf? What are the lines that contain
"pam_ldap.so" in your /etc/pam.d/system-auth and the other files in
that directory?
> Sometimes 1 or 2 services on these servers sucks 100% cpu and the load
> becomes high on the server.
Only Apache or other daemons as well?
> Below is an example where one the httpd process was eating 100% cpu
> and we took dump of this process
Do you have any LDAP authentication configured in Apache? Or any other
kind of authentication (PAM? System?) that might end up being served
by LDAP? Do you have an application, such as a PHP application that
would run inside an Apache process, that might be using LDAP?
> #0  0x00002ad1849cd997 in ldap_chase_v3referrals () from
/usr/lib64/libldap-2.3.so.0
Looks like it's getting in a loop of referrals, but it's hard to tell
for sure from one backtrace only.

You could try to get several backtraces and see if it's all the time
in that same function, that might indicate a loop.

Can you get a log of queries that the LDAP server is receiving (if it
is receiving LDAP queries at all while your process is in that loop)?
Can you use tcpdump to determine if you get a lot of LDAP traffic and
if the traffic stops when you kill the process?

Can you see what that Apache process was serving at that time, using
/server-status or something like it? That might give you a clue of why
the problem appeared.
> /etc/ldap.conf file:
> [...]
> timelimit 0
> sizelimit 0
Did you try to increase those?
> There are 2 bugs listed on the redhat site but no solution for this
> problem has been provided.
> https://bugzilla.redhat.com/show_bug.cgi?id=222667
> https://bugzilla.redhat.com/show_bug.cgi?id=474181
These do not seem related to your problem, as they report processes
that hang in a deadlock, which is not your case. If that would have
been your case, the process would be using 0% CPU instead of 100% CPU.

HTH,
Filipe