freebsd stable - Jan 2013 - FreeBSD 9.1 - openldap slapd lockups, mutex problems

Hi.

(Im am sending this to the "stable" list, because it maybe kernel
related.. )

On 9.1-RELEASE I am witnessing lockups of the openldap slapd daemon.

The slapd runs for some days and then hangs, consuming high amounts of CPU.
In this state slapd can only be restarted by SIGKILL.

 # procstat -kk 71195
  PID    TID COMM             TDNAME           KSTACK                       
71195 149271 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d do_wait+0x678
__umtx_op_wait+0x68 amd64_syscall+0x546 Xfast_syscall+0xf7
71195 194998 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _cv_wait_sig+0x12e
seltdwait+0x110 kern_select+0x6ef sys_select+0x5d amd64_syscall+0x546
Xfast_syscall+0xf7
71195 195544 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 196183 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_timedwait_sig+0x19 _sleep+0x2d4 userret+0x9e
doreti_ast+0x1f
71195 197966 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 198446 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 198453 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 198563 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 199520 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 200038 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 200670 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 200674 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 200675 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 201179 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 201180 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 201181 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 201183 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7
71195 201189 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d
_do_lock_umutex+0x5e8 do_lock_umutex+0x17c __umtx_op_wait_umutex+0x63
amd64_syscall+0x546 Xfast_syscall+0xf7

When I try to stop slapd through the rc script I can see in the logs that the
process is waiting for a thread to terminate - indefinitely.
Other multithreaded server processes running on the server without problems
(apache-worker, mysqld, bind, etc.)
On UFS2 slapd runs fine, without showing the error.


Things I have tried already to stop the lockups:

- running openldap-server23, openldap24 both with different BDB backend
versions.
- tuning the BDB Init File
- reducing the threads used by slapd through slapd.conf

What I didn't try until now:

Mounting a zfs vdev into the jail, to have the BDB storing its data on UFS.
(don't like the idea)


Environment:

- freebsd 9.1-rel-amd64 multijail server with cpu resource limit patch[1], which
didn't make it into 9.1-rel
- filesystem: zfs-only, swap on zfs
- active jail limits through rctl.conf (memory, maxprocs, open files)
- a handfull of openldap-server jails that show the same slapd lockup tendency.
- slapd started through daemontools (supvervise)

Some ideas:
- openldap-server with BDB backend uses sparse files for storing the data - on
top of ZFS.

Has anyone else running openldap-server on FreeBSD 9.1 inside a jail seen
similar problems?
How can I debug this further?

Any hints appreciated :-)

Regards.


[1] https://wiki.freebsd.org/JailResourceLimits

On 01/22/13 05:19, Kai Gallasch wrote:
> Hi.
>
> (Im am sending this to the "stable" list, because it maybe kernel
related.. )
>
> On 9.1-RELEASE I am witnessing lockups of the openldap slapd daemon.
>
> The slapd runs for some days and then hangs, consuming high amounts of CPU.
> In this state slapd can only be restarted by SIGKILL.
>
>   # procstat -kk 71195
>    PID    TID COMM             TDNAME           KSTACK
> 71195 149271 slapd            -                mi_switch+0x186
sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d do_wait+0x678
__umtx_op_wait+0x68 amd64_syscall+0x546 Xfast_syscall+0xf7
> On UFS2 slapd runs fine, without showing the error.
> Has anyone else running openldap-server on FreeBSD 9.1 inside a jail seen
similar problems?
I have seen openldap spin the cpu and even run out of memory to get 
killed on some of our test systems running ~9.1-rel with zfs.  No jails.
I'm not sure what would have put load on our test systems other than 
nightly scripts.  I had to focus my attention on other servers so I 
don't have one to inspect at this point, but I won't be surprised if I 
see this in production.  Thanks for the tip about it being ZFS related, 
and I'll let you know if I find anything out.  This is mostly a "me
too"
reply.

Am 22.01.2013 um 11:19 schrieb Kai Gallasch:
> Hi.
> 
> (Im am sending this to the "stable" list, because it maybe kernel
related.. )
> 
> On 9.1-RELEASE I am witnessing lockups of the openldap slapd daemon.
> 
> The slapd runs for some days and then hangs, consuming high amounts of CPU.
> In this state slapd can only be restarted by SIGKILL.

short update:

I tried all I could to isolate the problem.

What I am certain of is that the problem lies in the BDB backend for openldap
itself, or how slapd interacts with it.
My knowledge is not sufficient to debug BDB itself, it appears to be quite a
complex gearbox.

Also - as a sidenote - I had to learn that the new owner of BDB (orcle) does not
give a toss about keeping old links to BDB documentation intact (for example
informattion you'd need to tune your BDB - "DB_CONFIG" - or
understand it better.)

In the end I decided to drop the BDB backend for my running slapd installations
and switch over to MDB[1,2] as backend and since then the problems disappeared.

So if you are plagued by the same slapd lockups and rely on your ldap directory,
switching backends will give you some peace of mind.

Thanks to all who have replied! And thanks to sleepycat for all the fish.

Kai Gallasch.


[1] http://manpages.ubuntu.com/manpages/precise/man5/slapd-mdb.5.html
[2] http://www.openldap.org/doc/admin24/backends.html